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EXECUTIVE  SUMMARY 

The  National  Training  Center  (NTC)  was  instituted  in  1979  at  Ft.  Irwin,  California  to 
provide  a  central,  realistic  training  facility  for  combined  arms  combat  training.  The  NTC  was  given 
two  missions:  a)  to  provide  realistic,  essential  combined  arms  training;  and,  b)  to  gather  data  on  the 
effectiveness  of  current  and  emerging  doctrine,  force  structure,  organization,  materiel  systems  and 
training  management  approaches  (Edwards,  1979).  Units  began  arriving  at  the  NTC  for  training 
in  late  1981.  Two  additional  Combat  Training  Centers  (CTC)  were  added  during  years  1987  to 
1988;  Joint  Readiness  Training  Center  (JRTC)  and  the  Combat  Maneuver  Training  Center 
(CMTC).  At  the  inception  of  NTC,  deriving  information  and  data  was  integral  to  obtaining  the  full 
value  for  the  Army.  However,  due  to  start-up  difficulties  and  organizational  changes,  the  second 
mission  was  not  addressed  until  1985. 

In  1985,  Combined  Arms  Training  Activity  (CAT A)  contracted  with  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  at  the  Presidio  of  Monterey,  California 
(ARI-POM)  to  begin  the  process  of  extracting  information  fi'om  the  data  generated  at  the  NTC 
(LOA,  1985).  The  CTCs  generated  data  through  their  instrumentation  systems,  observations 
provided  during  training,  and  feedback  provided  for  home-station  training.  The  data  were  not 
generated  from  an  analytical  perspective,  but  solely  from  a  feedback  and  training  perspective. 
While  these  data  sources  were  merely  by-products  of  the  training  effort,  they  provided  the  Army 
with  an  opportunity  to  extract  information  to  assess  combat  readiness,  needed  training 
improvements,  and  effectiveness  of  new  equipment  and  organizations.  ARI-POM,  in  conjunction 
with  the  BDM  Corporation,  developed  and  brought  to  an  operational  status  a  CTC  research 
database,  named  the  CTC  Archive.  As  technology  advanced,  the  database  likewise  evolved.  Data 
sources  were  reviewed,  improved,  and  integrated.  Prototype  reports  and  responsive  research  issues 
were  produced.  Data  collection  and  measurement  approaches  were  developed,  proposed  and  tested 
to  facilitate  both  feedback  to  the  units  and  assessment  of  unit  combat  readiness.  Instruction,  tools, 
and  interactive  computer  programs  were  developed  to  enable  military  analysts  the  capability  to  use 
the  CTC  Archive. 

Over  the  period  of  ten  years,  ARI  documented  and  developed  systematic  data  collection  and 
analyses  which  provided  the  Army  with  lessons  learned,  trend  analyses,  and  the  tools  for  students 
and  trainers  to  extract  data  from  the  NTC  database.  The  result  of  CATA's  relationship  with  ARI- 
POM  is  a  fully  operational  data  collection  and  analytical  system,  complete  with  state  of  the  art 
databases,  training  workshops,  and  workstation  instructional  documentation.  The  NTC's,  now 
CTCs',  second  mission  to  provide  vital  and  responsive  information  to  the  Army  was  realized.  In 
1995,  ARI-POM  turned  over  to  TRADOC  a  fully  operational,  user  friendly  research  database  which 
satisfied  both  the  military  and  research  community  research  requirements. 

The  purpose  of  this  document  is  to  report  developments  towards  meeting  the  second 
mission.  The  following  is  a  documentation  of  the  database  development  and  what  eventually 
became  known  as  the  CTC  Archive  which  played  a  critical  role  in  achieving  the  Army  wide  lessons 
learned  mission.  The  information  here  serves  several  purposes:  a)  to  share  what  has  been  learned 
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through  the  process  of  creating  a  useable  database;  b)  to  outline  what  information  was  available 
through  the  CTC  Archive;  and  c)  to  document  how  military  analysts  and  researchers  have  used  the 
CTC  Archive  to  develop  lessons  learned  for  the  Army. 
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CHAPTER  I 

TRADOC  EFFORTS  TO  DERIVE  ARMY-WIDE  LESSONS  LEARNED 

1979-1985 

The  Combat  Training  Centers  (CTCs)  provide  realistic  combat  environments  by  employing 
simulated  weaponry  and  simulated  battlefield  effects.  These  battlefields  are  as  realistic  as  they 
possibly  can  be  without  the  loss  of  life.  CTCs  are  learning  environments  which  provide  units  a  test 
of  their  combat  readiness  and  feedback  to  guide  unit  training.  Early  on,  the  Army  recognized  the 
need  to  objectively  measure  the  effectiveness  and  efficiency  of  organizations  and  weapon  systems. 
The  CTC  developers  realized  the  unique  opportunity  to  provide  not  only  units  combat  experience, 
but  also  institutional  feedback  to  the  Army  (Chapman,  1992;  Edwards,  1979;  GAO  Report  1986). 

The  Development  Plan  clearly  outlined  a  procedure  and  requirement  to  develop  a 
standardized  evaluation  of  performance  and  effectiveness,  thereby  providing  a  resource  for  both 
analysis  and  feedback  (Edwards,  1979).  However,  this  requirement  was  not  met.  Institutional 
feedback  for  the  Army  was  superseded  by  the  need  to  provide  unit  feedback.  Hence,  institutional 
feedback  had  to  be  derived  from  data  generated  for  units. 

The  Combined  Arms  Center  (CAC)  and  the  combined  arms  schools  were  given  the 
responsibility  to  develop  analytical  measures  in  coordination  with  the  instrumentation  development 
conducted  by  HQ  TRADOC  (Edwards,  1979).  The  NTC  Database  was  to  be  maintained  to  provide 
FORSCOM  and  TRADOC  information  required  to  assess  combat  readiness,  develop  and  assess 
training  programs  and  doctrine,  assess  force  development  and  weapon  requirements,  and  to  improve 
combat  simulations  (Edwards,  1979).  While  the  core  instrumentation  system  (CIS)  was  designed 
with  the  ability  to  electronically  track  players,  weapons,  and  vehicles,  data  collection  provisions  for 
analysts  were  not  included  (Chapman,  1992;  GAO  Letter,  1986). 

The  National  Training  Center  (NTC)  at  Ft.  Irwin,  California,  the  first  combat  training 
center,  was  instituted  in  1979.  It  was  designed  with  a  laser  based  instrumentation  system  which 
simulated  casualties  and  firings  to  create  a  realistic  battlefield  environment.  The  instrumentation 
system  was  also  designed  to  track  soldiers  and  vehicles  as  the  battle  occurred.  Part  of  the  training 
experience  was  to  provide  feedback  during  the  training  events.  Information  gathered  electronically 
and  from  the  Observer/Controllers  (OCs)  was  designated  by  the  development  plan  as  the  major 
input  into  the  feedback,  termed  After  Action  Reviews  (AARs).  A  portion  of  the  Operations  Group, 
the  Training  Analysis  and  Feedback  (TAF)  division,  was  to  synthesize  this  information  with  video 
and  communications  information  into  an  AAR  package  for  each  mission  (Edwards,  1979).  Start-up 
difficulties  necessitated  a  different  approach  which  permanently  influenced  the  production  of  AARs 
and  impacted  the  generation  of  data  for  analysts. 

When  the  first  Army  maneuver  battalions  arrived  at  NTC  for  training  in  1981,  the 
instrumentation  system  was  still  suffering  start-up  difficulties  (Chapman,  1992).  The  CIS  was 
initially  conceived  as  an  integral  part  of  the  process  of  developing  AARs  for  units  during  training. 
AARs  were  part  of  the  "train-evaluate-train"  system  of  the  NTC  (Edwards,  1979).  After  missions 
were  conducted,  units  from  the  very  lowest  echelon  to  the  highest  echelon  were  provided 
information  on  their  performance,  both  strengths  and  weaknesses  (Chapman,  1992).  Many 
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problems  existed  with  the  instrumented  data  collection  system  which  spanned  the  continuum  from 
players  and  vehicles  which  were  uninstrumented  and  therefore  could  not  be  tracked,  to  the  system's 
inability  to  track  concealed  vehicles  and  personnel,  to  total  breakdown  of  the  CIS  (Chapman,  1992; 
Shackelford,  1985). 

Originally,  the  CIS  information  was  to  be  processed  by  the  TAF  division.  The  TAF 
consisted  of  two  training  analysis  teams:  one  designated  to  analyze  armor  battalion  task 
performance,  and  the  other  to  analyze  the  mechanized  infantry  battalion  task  force.  These  training 
analysts  were  initially  to  provide  all  the  training  feedback  needed  for  the  AARs  using  the 
instrumented  data  (Chapman,  1992;  Edwards,  1979).  However,  the  instrumentation  system  was 
unable  to  provide  information  for  AARs  during  the  first  year  of  rotations.  Thus,  the  OCs  began 
conducting  the  AARs  because  they  were  closest  to  battlefield  activities.  This  was  well  beyond  the 
original  scope  of  duties,  responsibilities  and  intent  for  the  OCs.  The  OCs'  envisioned  role  had  been 
limited  to  battlefield  operations,  such  as  providing  battlefield  effects,  assisting  in  the  casualty 
assessment,  and  enforcing  the  engagement  rules  and  Multiple  Integrated  Laser  Engagement  System 
(MILES)  procedures.  The  OCs  began  conducting  the  AARs  using  their  observations  on  the 
battlefield  as  well  as  their  knowledge  of  doctrine  (Chapman,  1992). 

OCs  worked  with  units  at  all  echelons  in  the  field  to  provide  guidance  and  remained  with 
the  units  throughout  the  rotation.  While  they  followed  the  battle  and  advised  units  on  how  to 
improve  their  battle  tactics,  they  compiled  subjective  observations.  At  the  end  of  missions,  they 
constructed  and  presented  AARs  with  information  gathered  from  the  instrumentation  (when 
available),  video  cameras  on  the  field,  and  their  subjective  observations.  Significant  events  which 
affected  the  battle  outcome  were  isolated  for  review.  Performance  strengths  and  weaknesses  were 
highlighted  and  reviewed.  At  the  end  of  the  rotation.  Take  Home  Packages  (THPs)  were  compiled 
from  all  the  AARs  and  eventually  data  gathering  instruments  during  the  rotation  and  sent  to  home 
stations.  THPs  became  a  primary  data  source  for  the  CTC  Archive. 

Colonel  Shackelford,  chief  of  operations,  designed  a  system  of  check  lists  to  facilitate  and 
standardize  the  feedback  of  OCs.  From  these  notes  and  input  from  the  OCs,  THPs  were  generated. 
The  checklists  did  not  standardize  the  AARs  however.  Banks  and  Whitmarsh  (1984)  reviewed  the 
AARs  and  Take  Home  Packages  and  reported  that  the  OCs  varied  on  types  and  methods  of 
observation  and  feedback.  This  hard  copy  data  was  initially  the  only  reliable  data  source,  though 
not  objective.  Even  after  the  instrumented  (CIS)  data  became  available  for  training  evaluation  and 
AARs,  these  data  alone  proved  unsatisfactory.  OCs  were  able  to  'correct'  instrumented  data  errors 
and  determine  causes  of  mission  failures  and  successes  through  observation.  Therefore,  OCs 
continued  to  present  the  AARs  complemented  by  the  instrumented  data  produced  by  the  Training 
Analysis  and  Feedback  Officer  (TAFO)  (Chapman,  1992;  Shackelford,  1985). 

Once  the  training  support  components  were  in  place  and  multi-echelon  combined  arms 
training  was  conducted  on  a  routine  basis,  emphasis  was  placed  on  NTC's  potential  for  addressing 
questions  concerning  training  techniques,  equipment,  organizations  and  doctrine  (Fobes,1984).  By 
1984  the  instrumentation  system  was  fairly  reliable  and  the  generation  of  THPs  consistent.  A 
requirement  to  establish  an  R&D  Center  which  had  the  capability  to  use  NTC  instrumented  and 
non-instrumented  data  for  analysis  emerged. 
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Although  from  inception  the  NTC  was  to  provide  Army  wide  lessons  learned,  the  ability 
and  commitment  to  do  so  was  absent.  As  noted  earlier,  the  NTC  Development  Plan  included  a 
directive  to  emplace  a  lessons  learned  system  at  the  NTC,  however  the  system  did  not  materialize. 
Brigadier  General  Frederic  J.  Brown,  III,  while  serving  as  Deputy  Chief  of  Staff  for  Training  at 
TRADOC  headquarters  in  1981,  directed  that  a  plan  be  defined  and  developed  for  using  the  data 
generated  at  the  NTC  for  Army  wide  institutional  feedback  two  months  prior  to  the  first  rotation. 
Although  he  recommended  the  establishment  of  a  group  of  representatives  from  TRADOC's  major 
subordinate  elements  to  define  and  develop  a  ‘Lessons  Learned’,  system,  no  group  or  system  was 
established  (Chapman,  1992). 

Brigadier  General  Frederick  J.  Brown,  III  designated  CAC  as  the  lead  agency  to  develop 
lessons  learned  for  the  Army.  CAC  responded  by  delegating  that  responsibility  to  the  NTC  Division 
of  the  Command  and  General  Staff  College's  (CGSC)  Unit  Training  Support  Directorate.  CGSC 
was  responsible  for  repositing  NTC  data  and  observations  and  analyzing  the  information  for  lessons 
learned  until  1984  when  the  CATA  was  established  (Chapman,  1992). 

As  late  as  1983,  the  problem  of  deriving  systematic,  objective  lessons  learned  continued. 
The  NTC  was  primarily  concerned  with  training  difficulties  which  resulted  from  the  limited  number 
of  trained  staff  (OPFOR)  and  instrumentation  difficulties.  Apparently,  the  issue  of  a  feedback 
mechanism  to  the  Army  was  over-ridden  by  training  concerns.  Major  General  Fredrick  J.  Brown  III 
was  commanding  the  Armor  school  at  the  time  and  raised  the  issue  of  processing  data  from  the 
NTC  at  a  meeting  of  branch  proponent  school  representatives.  However,  no  actions  were  taken,  nor 
any  concrete  system  designed  nor  implemented  to  extract  information  for  the  Army  from  the  NTC. 
Throughout  the  history  of  the  NTC,  a  persistent  concern  and  resistance  regarding  evaluation,  or 
"tests",  at  the  NTC  has  thwarted  efforts  towards  any  objective  measurement  of  unit  performance. 
FORSCOM  unit  commanders  worried  that  their  performance  at  the  CTCs  would  adversely  affect 
their  assignments  and  promotions,  and  therefore  resisted  any  objective  measurement  of  unit 
performance  (Chapman,  1992). 

While  the  Army  and  TRADOC  proponents  continued  to  be  concerned  about  using  the  NTC 
data  to  the  fullest,  in  1984  there  still  existed  no  method  for  integrating  the  data  into  one  database  for 
analysis,  or  a  systematic  method  of  generating  lessons  learned.  CAC  published  lessons  learned  for 
battalion  task  force  commanders  and  staff  under  the  title  of  "Combined  Arms  Training  Tips" 
(Chapman,  1992),  but  these  were  largely  anecdotal.  These  lessons  learned  were  based  primarily  on 
commander  insights  and  OC  observations  at  the  NTC.  The  data  were  not  objective,  nor  were  the 
Combined  Arms  Training  Tips  systematically  derived. 

In  1984,  Army  Research  Institute  (ARJ)  was  contacted  by  TRADOC  to  assist  in  the 
development  of  NTC  data  utilization.  Initial  research  products  generated  by  ARJ  at  the  Presidio  of 
Monterey  (ARI-POM)  pertained  to  the  contents  and  tools  with  which  to  examine  the  data 
(Dunipace,  Henderson,  &  Smart,  1984;  Fobes,  1984;  McFann  &  Avant,  1983;  Weaver,  Griesmer, 
&  Alan,  1984). 

ARI's  involvement  with  CAC  provided  TRADOC  the  ability  to  respond  to  a  General 
Accounting  Office  (GAO)  investigation  which  questioned  the  cost-effectiveness  of  the  NTC  during 
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FY  84-85  (GAO  report,  1986).  The  report  cited  the  extensive  turbulence  within  units  after  NTC 
rotations  as  a  reason  to  shift  from  exclusive  emphasis  on  training  feedback  and  to  begin  collecting, 
analyzing,  eind  processing  lessons  learned  into  the  Army's  institutional  memory.  At  that  time, 
except  for  ARI  and  CAC  plans,  no  work  was  being  conducted  on  the  NTC's  second  mission,  to 
provide  information  from  the  NTC  regarding  systemic  Army  problems  (Banks,  1993;  Chapman, 
1992). 


In  August  of  1985  another  effort  was  made  to  institutionalize  a  lessons  learned  system 
(Chapman,  1992).  A  Center  for  Army  Lessons  Learned  (CALL)  was  established  at  CAC  as  a 
directorate  of  the  CATA.  In  an  effort  to  better  manage  the  NTC  data  collection,  the  Army  also 
established  a  Data  Analysis  Center  at  ARI-POM  and  a  Combat  Analysis  Laboratory  at  the  RAND 
Arroyo  Center  at  Santa  Monica,  California. 

CATA  formally  contracted  with  the  ARI  in  1985  (LOA,  1985)  to  establish  an  R&D  center 
to  explore  the  data  generated  at  the  NTC.  ARI-POM  planned  and  directed  a  multimillion  dollar 
Unit  Training  R&D  Center  to  provide  the  initial  research  capability  and  analytic  tools  to  use  the 
massive  amounts  of  data  generated  during  NTC  operations.  The  commitment  by  the  Army  to 
derive  as  much  as  possible  from  the  NTC  became  substantial. 

Acting  Commander  of  the  CATA,  Colonel  Jerome  L.  Haupt  signed  an  agreement  with  the 
Commander  of  the  ARI,  Colonel  W.  Darryl  Henderson  on  16  September  1985.  In  this  LOA,  ARI 
was  directed  to  begin  addressing  the  CTCs'  second  mission.  ARI  was  to  develop  and  prototype 
methods  and  procedures  to  assess  unit  training  status,  develop  NTC  training  and  feedback  systems, 
develop  standardized  procedures  for  generating  NTC  THPs,  and  provide  recommendations  for  the 
NTC  data  base  design  and  application  (LOA,  1985). 
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CHAPTER  II 

INITIAL  DERIVATION  OF  LESSONS  LEARNED 

In  1985  ARI  was  awarded  $1.5  million  dollars  for  a  3-year  effort  to  develop  performance 
measures,  evaluate  the  usefulness  of  NTC  instrumented  data  for  analyzing  exercise  results,  and 
assist  in  developing  an  analysis  methodology.  In  the  first  year,  22  research  products  and  reports 
were  produced  (Doherty,  1987).  These  products  were  developed  in  support  of  ARI's  effort  to 
investigate  new  methodologies  for  measurement  of  unit  performance  and  the  derivation  and 
dissemination  of  lessons  learned.  The  BDM  corporation,  a  military  contractor,  was  contracted  to 
assist  in  a  three  year  investigation  entitled:  "Research  Support  for  a  Unit  Home  Station  Training  and 
Feedback  System".  BDM  played  a  major  role  throughout  the  development  and  operation  of  the 
CTC  Archive. 

McFann  and  Avant  (1983)  explored  the  data  generated  at  the  NTC  to  determine  what  data 
sources  existed  and  their  availability  for  potential  use  by  the  Army  research  community.  McFann 
and  Avant  (1983)  concluded  that  before  the  NTC  digital  database  could  of  any  use,  it  needed  to  be 
assessed  for  fidelity  of  the  simulation,  the  extent  and  nature  of  the  simulation,  the  amount  of 
erroneous  or  lost  data,  the  extent  of  pre-editing  of  the  data,  and  the  actual  battlefield  situation.  After 
the  signing  of  the  LOA  in  1985,  the  very  first  steps  towards  establishing  the  R&D  center  were  to 
examine  both  the  types  and  quality  of  data  available  at  the  NTC.  Systematic  processing  of  the  data 
was  required  before  the  research  community  could  effectively  use  the  data  generated  at  the  NTC. 

Whitmarsh  (1987)  conducted  a  study  examining  the  types  and  quality  of  the  NTC  data.  He 
examined  the  types  of  NTC  information  available,  research  problems  regarding  collection  and 
merging  of  information,  and  data  quality  issues.  The  data  sources  he  found  available  were  the 
Operations  Plans,  Take  Home  Packages  (THPs),  the  Company/Team  THPs,  video  taped  After 
Action  Reviews  (VAARs),  Unit  After  Action  Reports  (UAARs),  and  the  NTC  Instrumentation 
System  (NTC-IS)  Data  Tapes.  Later  the  Communications  Tapes  which  recorded  battlefield 
transmissions  were  identified  and  also  sent  to  ARI-POM.  The  research  problems  he  identified  were 
in  the  collecting  and  merging  of  the  data.  All  the  various  sorts  of  data  required  integration  to 
provide  an  effective  data  source  for  any  analytical  effort.  No  one  data  source  alone  could  be 
considered  reliable,  many  errors  and  omissions  occurred  due  to  a  lack  of  collection  standardization. 
Overlap  and  redundancy  in  the  various  data  sources  permitted  correction  of  erroneous  data, 
identification  of  missing  data,  and  a  more  realistic  and  encompassing  view  of  battlefield  activities. 

At  the  end  of  the  initial  phase  of  the  NTC  Archive,  data  were  routinely  collected  and 
forwarded  to  ARI-POM.  The  archive  contained  eight  years  of  data  from  Armor  and  Mechanized 
Infantry  Task  Force  rotations  beginning  with  the  first  NTC  rotation  (8101).  However,  the  data 
beginning  fiscal  1986  were  more  reliable  than  previous  years  because  sufficient  standardization  of 
data  collection  and  rules  of  engagement  had  been  emplaced.  In  order  to  facilitate  the  use  of  the  CTC 
Archive,  workshops  were  instituted  at  ARI-POM  to  provide  analysts  the  tools  and  knowledge 
necessary  to  derive  lessons  learned.  The  data  sources  identified  and  processed  by  this  time  are 
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listed  in  Table  1'  (Zimmerman,  1989). 

In  an  effort  to  integrate  these  various  data  sources  and  formats,  ARI-POM  proposed  a 
system  which  would  include  all  these  elements.  This  research  database  was  largely  realized  with  the 
exception  of  integrating  the  VAAR,  UAARs,  or  communications  tapes  into  the  digital  database.  In 
the  initial  stages  of  beginning  to  process  NTC  generated  data,  only  the  THPs  and  the  digitized  CIS 
tapes  could  be  processed  into  an  electronic  database.  The  NTC-IS  Data  Tapes  were  entered  by  tape 
drive  through  a  NTC  Translator/Loader  software  by  a  computer  system  operator.  THPs  were  added 
through  the  word  processing  system.  Retrieval  of  the  information  was  complex  and  difficult 
because  the  instrumentation  system  was  not  designed  with  an  analytical  perspective  (Whitmarsh, 
1987). 


As  early  as  the  first  year  of  taking  on  responsibility  for  developing  an  archive,  ARI-POM 
was  deriving  lessons  learned  fi'om  the  available  NTC  generated  data  (Shackelford,  1985).  The  data 
sources  used  for  this  study  were  the  THPs,  the  UAARs  and  the  CIS  data  tapes.  The  data  were 
derived  from  14  battalions  which  rotated  through  the  NTC  from  1982  to  1984.  Areas  examined 
were  task  force  performance,  main  battle  tank  loss  rates,  tank  and  TOW  ammunition  expenditures 
compared  to  historical  expenditures,  effectiveness  of  direct  fire  during  defensive  missions,  NTC 
combat  vehicle  loss  ratios,  live-fire  tank  gunnery  performance,  and  task  force  communications 
security  performance.  Although  the  data  sources  were  stable  during  the  time  period  under  study, 
the  reliability  of  the  data  had  to  be  considered.  Nonetheless,  with  good  military  judgement  and 
reasonable  data  sources,  unprecedented  studies  emerged  (Shackelford,  1985).  The  initial 
development  of  the  NTC  database  provided  both  trainers  and  doctrine  developers  with  a  much 
needed  resource. 

Once  the  data  sources  had  been  identified  and  explored  as  to  content,  an  effort  was  made  to 
develop  short  term  studies  to  test  the  data  sources.  The  immediate  value  to  the  Army  was  becoming 
apparent.  Short-term  studies  included  examining  the  effect  of  new  equipment  and  organizations  to 
determine  the  effectiveness  of  the  new  developments  in  simulated  environment. 


’For  a  more  detailed  description  of  the  development  of  data  sources,  see 


Chap.  3  "Data  Sources  &  Development". 
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Table  1.  Data  Available  to  Analysts  in  1988. 


INGRES  Digital  Data:  This  NTC  digital  database  was  derived  from  both  the  CIS  log  and 
RDMS  log  digital  data.  The  database  consisted  19  mission-level  INGRES  tables  and  the 
event  driven  data  captured  during  the  engagement  training  exercises. 

Contour  and  Player  Location  Plots:  Printouts  illustrated  the  elevation  contours  and 
player  locations  at  a  specified  time  during  the  battle.  These  were  derived  from  the  digital 
database  in  conjunction  with  plots  developed  on  the  VAX  computer. 

DeAnza  Graphics  Display  and  VT125  Data  Display:  Digitally  recorded  battle 
playback  and  information  about  the  events  as  they  occurred  could  be  obtained  through  the 
DeAnza  Replay  Stations. 

General-Purpose  NTC  Analysis  of  Training  Tool:  A  software  tool  developed  at 
ARI-POM  allowed  player  movement  and  engagements  to  be  viewed  and  analyzed  on  any 

IBM  PC  or  compatible  computer. 

Take  Home  Packages:  Feedback  to  the  units  by  BOS  by  Co/Tm  and  Task  Force  were 
documented  along  with  some  battle  statistics. 

Communications:  All  radio  communications  were  recorded  and  stored  on  tape. 

Video  After  Action  Reviews:  AAR  video  recordings  were  available  for  Task  Force, 
company/team,  platoon.  Fire  Support  Battalion,  Field  Artillery,  and  Live  Fire  missions. 

Initialization  File:  A  hard  copy  document  with  information  on  instrumented  unit  and 
player  listings,  primarily  used  by  database  managers  to  correlate  RDMS  data  with  CIS  data. 

Operations  Orders:  Documents  which  described  the  scenarios  developed  for  unit  training 
at  the  NTC. 

Combined  Arms  Assessment  Team  Reports:  Contained  summary  reports  of  SME 
observations  during  NTC  Focused  Rotations. 

Standing  Operating  Procedure:  Administrative,  logistical,  and  training  procedures  used 
for  combat  processes  were  available. 

NTC  Rules  of  Engagement:  A  document  which  contained  the  basic  guidelines  for  the 

ESX  and  LFX  training  at  the  NTC.  The  ROE  applied  to  both  the  OPFOR  and  visiting  units. 

Daily  Staff  Journals:  Unit  logs  of  activities  by  time  and  event  kept  by  operators  at  each 
unit. 

Unit  After  Action  Reports:  Hard  copy  reports  were  generated  by  units  and  included  a 
summary  of  their  perceptions  of  training  events  and  unit  performance. 
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Nichols  (1986)  at  ARI-POM  conducted  a  study  using  the  THPs  as  a  data  source.  She 
analyzed  the  efficacy  of  the  H-Series  as  opposed  to  the  J-Series  MTOE,  overall  unit  performance, 
and  mission  performance  of  Armor  versus  Mechanized  Task  Forces  (TF)  during  Force-on-Force 
simulated  battle.  This  study  demonstrated  the  utility  as  well  as  the  shortcomings  of  THPs  as  a  data 
source.  THPs  were  found  to  be  reliable,  but  limited  since  they  contained  only  macro  level  type 
variables.  The  study  reported  an  advantage  by  the  Mechanized  TF  rmder  the  J-Series.  This 
advantage  was  demonstrated  only  during  offensive  engagements.  The  Armored  TFs  showed  only 
minimal  advantage  during  either  defensive  or  offensive  missions  under  the  J-Series.  While  these 
results  were  useful,  a  broader,  more  comprehensive  data  source  was  required  to  determine  the  cause 
or  reasons  for  these  differences.  The  results  of  this  study  prompted  the  exploration  on  the  digital 
data  to  explain  the  findings.  Hence,  the  concept  of  a  multi-step  research  effort  was  developed 
(Nichols,  1986). 

Root  (1986)  used  all  available-data  sources  and  compiled  an  analysis  on  one  mission  for  one 
TF.  The  report  demonstrated  the  ability  of  the  combined  NTC  data  sources  to  provide  a  complete 
picture  of  battlefield  events.  Root  conducted  an  analysis  of  a  deliberate  attack  using  data  from  the 
VAX  Digital  Tape  Extract  derived  from  the  instrumented  data  which  presented  the  positions  of 
Co/Tms  graphically  at  the  critical  time  of  the  mission.  The  DeAnza  Extracts,  which  were  selected 
photographs  of  the  ongoing  battlefield  events,  showed  the  movement  of  the  TF  during  the  mission 
at  the  critical  moments.  THP  information  was  used  to  clarify  the  particular  details  only  accessible 
through  direct  observation  of  the  OCs.  THPs  also  provided  information  such  as  Battle  Damage 
Assessment  (BDA)  and  the  planning  and  preparation  activities  which  occurred  before  the  execution 
of  the  mission.  The  Communications  Tapes  were  listened  to  and  presented  in  written  form  for  the 
analysis.  The  Communications  Tapes  provided  insight  regarding  the  erroneous  firing  of  artillery 
which  revealed  the  error  was  a  transposition  of  numbers.  The  Communications  Tapes  also 
demonstrated  the  confusion  on  the  battlefield  and  within  the  TOC,  and  the  poor  reporting  by 
subordinates.  Although  time  consuming,  this  effort  demonstrated  the  detailed  insight  available 
from  the  NTC  database  which  could  facilitate  home-station  training.  Since  the  missions  conducted 
at  the  NTC  are  fairly  uniform  from  rotation  to  rotation,  conducting  an  in-depth  analysis  over  time 
could  provide  indicators  of  systemic  problems  for  Army  wide  lessons  learned. 

Heretofore  lessons  learned  were  generated  primarily  by  senior  staff  officers  and  chiefs  of 
operations  upon  their  departure  from  NTC.  The  issues  they  addressed  were  derived  subjectively, 
although  expertly.  One  potential  use  of  the  enormous  amounts  of  data  that  were  arriving  was  to 
develop  a  method  to  derive  issues  systematically  and  objectively.  Ritenour  (1986)  reviewed  the  two 
primary  feedback  sources,  THPs  and  AARs,  and  compiled  an  issues  document.  He  extracted 
systematically  those  comments  made  in  the  THPs  and  AARs  which  were  repeated  over  time.  He 
organized  these  comments  by  Battlefield  Operating  Systems  (BOS),  which  was  the  rubric  used  by 
OCs  during  the  conduct  of  AARs.  Roughly  315  issues  were  identified  which  could  be  examined 
with  the  NTC  data  generated  at  that  time.  Although  the  list  of  issues  was  not  exhaustive  nor 
perhaps  comprehensive,  the  study  demonstrated  how  the  NTC  data  could  be  used  by  the  Army  to 
focus  derivation  of  lessons  learned. 

In  addition  to  the  verification  of  the  importance  of  the  NTC  data  to  the  Army  through  the 
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conduct  of  these  early  studies,  ARI-POM  began  developing  a  methodology  for  measuring  imit 
combat  effectiveness  at  the  NTC  (Forsythe,  1986).  The  Army  had  long  intended  to  assess  combat 
effectiveness  during  peacetime  in  order  to  improve  training  and  doctrine.  The  Army  was  also  very 
interested  in  obtaining  feedback  on  equipment,  manning  and  organizational  structure.  While  the 
NTC  database  provided  great  quantities  of  data,  an  analytical  framework,  or  model,  was  required  to 
direct  the  data  collection  and  analysis  efforts.  A  methodology  was  developed  to  provide  the 
research  community  with  the  information  it  required  to  assess  unit  performance  and  to  provide  a 
link  between  NTC  results  and  home  station  training.  Mission  critical  events  were  explored  to 
determine  how  they  could  be  observed,  and  how  events  could  be  recorded  and  analyzed  through 
standards  of  performance,  measures  of  performance,  and  measures  of  effectiveness.  This  effort 
resulted  in  a  systematic  approach  to  measure  performance  and  potentially  permitted  a  more 
integrated  and  comprehensive  consideration  of  variables  which  systematically  affect  xmit 
performance.  The  accomplishment  of  specific  tactical  missions  was  the  focus,  not  the 
accomplishment  of  voluminous  AMTPs.  By  establishing  mission  effectiveness  criteria,  the 
identification  of  strengths  and  weaknesses  could  be  identified  over  time  and  provide  both  imit 
specific  and  Army-wide  lessons  regarding  training,  doctrine,  equipment  and  organization,  By 
providing  a  method  and  ultimately  a  tool  which  wed  both  the  scientific  and  military  priorities 
through  this  focus  on  performance-data  gathering,  meaningful  and  objective  lessons  learned  could 
be  provided.  This  study  and  those  that  spun  from  it  provided  tools  to  derive  institutional  lessons 
learned  from  the  NTC  (as  addressed  in  the  1986  GAO  letter). 


Two  major  ARI  projects  confirmed  the  feasibility  and  value  of  unit  performance 
measurement  of  combat  units  at  a  CTC  and  revealed  limitations  of  the  CTC  Archive.  The  first  was 
a  study  which  demonstrated  the  positive  correlation  between  operating  tempo  (OPTEMPO)  as 
measured  by  the  annualized  tank  mileage  employed  by  units  prior  to  their  rotation  and  performance 
at  NTC  as  measured  by  casualty/exchange  ratios  (Hiller,  McFann,  &  Lehowicz,  1990).  The  data  set 
for  NTC  performance  was  extensive:  100  missions,  during  16  rotations,  by  units  from  five 
installations.  A  set  that  extensive  was  stable  enough  to  determine  the  relationship  between 
OPTEMPO  and  unit  performance  capability.  The  results  were  cited  by  the  Secretary  of  Defense 
and  used  to  defend  the  Army  training  budget. 


The  second  project  to  consider  performance  at  a  CTC  was  Project  Determinants  (Holz, 
O'Mara,  &  Keesling,  1994).  This  project  studied  seven  rotations  and  assessed  the  success  of  each 
mission.  Since  there  was  no  credible  measure  of  success  that  could  be  applied  for  each  mission, 
ARI  researchers  coordinated  with  NTC  OCs  for  tailored  direct  assessments.  The  Archive  was  a 
resource  to  identify  recurring  problems  and  to  determine  the  scope  of  available  data.  The  results 
confirmed  Army  training  doctrine  in  general  and  the  previous  OPTEMPO  findings.  In  addition, 
ARI  researchers  were  able  to  use  NTC  data  to  address  issues  related  to  leadership,  cohesion,  and 
individual  characteristics.  Like  the  OPTEMPO  project.  Determinants  confirmed  the  feasibility  and 
usefulness  of  unit  performance  measurement  of  combat  units  at  NTC. 


By  1 987,  ARI-POM  was  producing  substantive  and  systematic  reports  on  all  the  topics 


11 


CTC  Archive 


described  in  the  LOA^.  The  following  is  an  excerpt  of  the  studies  conducted  at  this  time.  The 
topics  covered  are  noteworthy;  these  studies  addressed  doctrine,  equipment,  organization,  and 
combat  effectiveness  using  the  NTC  generated  data  (See  Table  2.).  The  conduct  of  these  studies 
demonstrated  the  utility  of  the  NTC  data,  as  well  as  the  ability  to  provide  information  to  the  Army 
so  that  it  could  begin  to  meet  the  mission  of  deriving  Lessons  learned  from  the  NTC. 


Table  2.  Extract  of  Initial  Studies 


National  Training  Center  Mortar  Fire  Support:  Relationship  to  Training  Doctrine.  Hamza  & 
Williams ,  1986. 

NTC  Performance  Trends  for  the  Maneuver  Operating  System:  Relationship  to 
Training  Doctrine.  Johnson ,  1986. 

A  Survey  Comparing  the  M2/M3  Bradley  Fighting  Vehicle  and  the  Ml  13  Armored 
Personnel  Carrier  BY  Members  OF  THE  NTC  Operations  Group  AND  OPFOR.  Hamza, 

1986. 

Direct  Fire  Fratricides  AT  THE  National  Training  Center.  Hamza  &  Banks,  1987. 
Commander  Survivability  AT  THE  National  Training  Center.  Doherty  &  Atwood,  1987. 
Commander  Survivability  at  the  National  Training  Center.  Zimmerman,  Brosemer,  & 
Hamza,  1987. 

Comparison  of  the  BFVs  and  Ml  13  Equipped  Battalion  Task  Forces  on  Live  Fire 
Performance  a  t  the  NTC.  Doherty  &  Hiller  ,1987. 

An  NTC  Live  Fire  Performance  Analysis.  Forsythe  &  Doherty,  1987. 

A  Preliminary  Analysis  OF  NTC  Force-on-Force  Performance.  Nichols,  1987. 

A  Method  of  Analysis  for  the  Bradley  Fighting  Vehicle  System.  Shackelford,  1987. 
Trendline  Analysis  OF  Direct  Fire  AT  THE  National  Training  Center.  Zimmerman,  1987. 


During  this  initial  period,  ARJ-POM  conducted  three  directions  of  research  all  pertaining  to 
the  exploitation  and  full  realization  of  the  NTC  data.  The  research  conducted  consisted  of  :  a) 
methodological  research  to  organize  and  analyze  training  data,  b)  analytical  research  on  high 
priority  training  issues  in  support  of  CALL,  c)  applied  research  to  develop  products  for  use  within 
the  Army  training  environment  (Lewman,  Zimmerman,  Briscoe,  Root,  &.  Allan,  1988).  In  1988, 
ARI-POM  began  producing  regular  trendlines  for  CALL  lessons  learned.  The  method  used  was 
systematic  and  derived  from  all  the  data  sources  available.  While  the  initial  reports  were  in 
response  to  immediate  Army  concerns  and  demonstrative  of  the  ability  to  derive  lessons  learned, 
later  studies  and  reports  demonstrated  the  capability  to  derive  long-term,  institutional  Lessons 
learned.  While  reports  and  instruments  were  developed  in  this  initial  time  period,  an  extensive 
effort  was  made  to  provide  military  analysts  access  to  the  data  (Lewman,  1988). 

In  1988,  Colonel  Kent  Harrison  was  commander  of  CALL,  newly  instituted.  His  job  was 
to  create  "an  agency  for  change".  He  formulated  working  groups  to  analyze,  collect,  and  process 


-  See  Appendix  A  for  a  Bibliography  of  studies  derived  from  the  CTC-Archive. 
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the  data  generated  at  the  CTCs  to  provide  input  into  Doctrine,  Training,  Organization,  Leadership, 
Materiel,  and  Soldiers  (DTOLMS).  Initially,  the  groups  began  identifying  issues  using  the  CTC 
Archive.  Whenever  an  issue  or  problem  continued  over  time  and  over  units  (i.e.,  occurred  8  times 
or  more),  it  was  identified  as  a  possible  "item".  These  items  were  then  compiled  into  a  huge 
laundry  list  of  potential  lessons  learned  and  submitted  to  CAC  leaders  and  the  relevant  proponents. 
The  proponents  then  selected  those  items  which  seemed  the  most  important  to  them.  Solutions 
were  created  by  the  proponents  and  implemented  into  one  or  more  units.  When  the  units  rotated  to 
a  CTC,  they  underwent  a  'focused'  rotation  which  examined  the  efficacy  of  the  change.  The  cycle 
from  submitting  the  items  to  the  implementation  of  a  solution  was  termed  an  Action  Plan. 

Hence,  the  cycle  became: 

1 .  Identify  issues  by  examining  CTC  Archive  Data. 

2.  Submit  issues  to  CAC  leaders  &  proponents. 

3.  Identify  issue(s)  for  action. 

4.  Derive  solution(s). 

5.  Implement  solution(s)  at  home  station  training. 

6.  Conduct  focused  rotation  to  examine  efficacy  of  solutions. 

7.  Implement  change  throughout  the  Army  or  derive  another  solution. 

An  example  is  the  issue  of  fratricide.  Units  rotating  through  the  CTCs  consistently  had  high 
fratricide  rates.  When  the  issue  was  submitted  to  the  schools  and  proponents,  they  realized  that 
units  trained  with  only  US  targets.  Enemy  and  US  targets  were  then  introduced  together  so  that 
units  would  have  to  discriminate  between  the  two  at  home  station.  During  the  focused  rotation  on 
fratricide,  units  which  trained  with  enemy  and  US  targets  had  fewer  fratricides.  Now  units 
regularly  train  with  both  US  and  enemy  targets. 
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CHAPTER  III 
TRADOC  STUDIES 

A  pivotal  event  in  the  transition  of  the  archive  from  an  R&D  asset  to  an  operational  resource 
was  a  directive  by  General  Thurman,  TRADOC  Commander,  for  "a  structured  exploitation  of  the 
NTC  database  resident  at  ARI-POM."  An  instructional  period  was  required  to  familiarize  military 
analysts  with  the  NTC  Archive  capability  and  tools  before  the  database  could  be  used.  ARI-POM 
developed  training  manuals  and  an  instruction  course  to  help  the  analysts  derive  the  information 
they  required.  Certification  of  users  also  helped  to  restrict  access.  This  led  to  a  series  of  seven  one- 
week  combined  arms  workshops  involving  90  people  from  10  schools  and  centers.  Each  workshop 
included  several  branches.  For  example,  the  Armor  workshop  was  supported  by  personnel  from  the 
Infantry,  Field  Artillery,  Engineer,  and  MI  Schools.  ARI-POM  conducted  a  prototype  workshop  in 
February  1988  and  then  hosted  the  combined  arms  workshops  from  March  to  Jime  of  that  year. 

The  lead  school  or  integrating  center  developed  issues  to  be  addressed  before  each 
workshop  and  CALL  reviewed  the  issues  with  ARI  and  NTC  for  feasibility.  The  first  day  of  the 
workshop  was  devoted  to  training  on  the  archive.  The  scope  included  an  overview  of  the  CTCs, 
discussion  of  issues  of  data  security  and  workshop  objectives,  a  description  of  data  sources  with  a 
discussion  of  strengths  and  weaknesses,  a  sample  case  study,  and  a  practical  exercise  that  required 
participants  to  execute  a  query  for  the  instrumented  data.  Participants  then  spent  the  remainder  of 
the  week  conducting  research  specific  to  the  issues  developed  by  the  lead  agency.  ARI  scientists, 
contractor  personnel,  and  CAT A/CALL  staff  assisted  with  the  research  phase. 

At  the  end  of  the  week,  each  team  presented  a  briefing  on  their  results  to  the  commandant 
of  the  lead  school  or  center.  The  issues  addressed  are  summarized  in  Table  3.  In  some  cases  the 
issues  listed  are  representative  of  a  longer  list.  All  agencies  also  commented  on  the  strengths  and 
weaknesses  of  the  data,  and  recommended  additional  information  to  be  collected. 

Each  proponent  attending  the  workshop  then  presented  the  results  in  a  briefing  to  General 
Thurman  in  June  1988.  All  proponents  considered  the  archive  to  be  valuable  despite  gaps  in 
information  and  the  difficulty  of  accessing  some  of  the  data  (most  frequently  digital  data  and 
communications  tapes).  The  CAC  presentation  at  this  briefing  presented  a  plan  to  move  the  CTC 
Archive  to  Fort  Leavenworth. 

Beginning  in  August  1988,  the  workshops  were  scheduled  on  a  quarterly  basis.  Completion 
of  the  training  was  a  pre-requisite  for  access  to  information  in  the  archive.  The  one-day  structured 
training  on  the  archive  continued  (the  practical  exercise  was  the  basis  for  certification),  but 
briefings  of  research  projects  were  not  required.  Much  of  the  impetus  in  the  first  three  quarterly 
workshops  were  efforts  to  augment  findings  for  an  update  briefing  to  General  Thurman  in  May 
1989. 
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Table  3.  Issues  Addressed  During  CTC  Archive  Workshops. 


Agency 

Selected  Issues  Addressed 

Infantry  School 

• 

• 

• 

• 

Contributions  of  Echo  Company 

Reconnaissance  requirements/capabilities  at  TF/Bde 

Contributions  of  mortars 

Mine  clearing  effectiveness 

Effectiveness  of  decision  making  process 

Logistics 

Adequacy  of  graphic  aids 

Armor  School 

• 

• 

• 

Relationship  of  tank  battle  participation  with  OPFOR/BLUEFOR  loss  ratio 

Relationship  between  number  and  quality  of  survivability  positions  and  friendly  force  survival 
Synchronization  of  CAS,  attack  helicopters,  artillery,  obstacles,  and  direct  fire 

Engineer  School 

• 

• 

• 

• 

Adequacy  of  breaching  doctrine 

Defensive  frontages 

Number  of  survivability  positions 

Fratricide  from  friendly  minefields 

Quality  of  obstacles  and  survivability  positions 

Field  Artillery 

• 

Rehearsal  of  fire  plan 

School 

• 

• 

• 

Fire  support  execution  matrix 

Location  of  TF  FSO 

Understanding  of  commander's  intent 

Length  of  target  list 

Massing  of  fires 

Intel  School 

• 

• 

• 

S2  role  in  staff  planning/wargaming  process 

Reconnaissance  and  surveillance 

Conterreconnnaissance/Countersurveillance 

Reporting;  Combat  information;  dissemination  of  intelligence 

GSR  employment 

MI  Co  team  operations  (influence  of  MI  unit  support) 

ADA  School 

Impact  of  ADA  on  freedom  to  maneuver  and  survivability 

Effectiveness  and  impact  of  early  warning 

Aviation  School 

• 

Aviation  IPB 

Aviation  unit  battlefield  movement 

Attack  helicopter  engagement  techniques 

Logistics  Center 

• 

Conclusions  on  doctrine,  training,  organization,  and  materiel: 

-  Arm 

-  Fuel 

-  Fix  (Maintain,  Class  IX,  Recover/Evacuate) 

-  Sustain  (Class  I,  Medical,  FSB) 

Signal  School 

• 

• 

Signal  planning  and  preparation 

Information  flow 

Soldier  Support 

• 

Operational  planning 

Center 

• 

• 

Reporting 

Mutual  support  between  S I  and  S4 

As  the  emphasis  from  TRADOC  waned,  the  quantity  and  quality  of  research  conducted  by 
military  analysts  decreased.  The  CALL  and  ARI  representatives  responsible  for  the  training 
concluded  that  the  limited  structure  did  not  lead  participants  to  master  enough  of  the  archive  tools 
to  receive  maximum  benefit  from  the  training.  However,  the  workshops  continued  to  be  offered  on 
a  quarterly  basis  with  fair  attendance  from  the  various  schools  and  military  branches. 
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CHAPTER  IV 

CTC  DATA  ACCESS  DEVELOPMENT 

Two  additional  Combat  Training  Centers  (CTC)  were  established  from  1987  to  1989;  the 
Joint  Regional  Training  Center  (JRTC)  at  Fort  Chaffee,  Arkansas  and  the  Combat  Maneuver 
Training  Center  (CMTC)  at  Hohenfels  Training  Area  in  the  Federal  Republic  of  Germany.  The 
data  produced  at  these  CTCs  were  reviewed,  catalogued  and  incorporated  into  the  NTC  database 
(Nichols,  1988;  Lewman,  Zimmerman,  Briscoe,  Root,  Allan  &  Shillcock,  1989).  Though  each  of 
the  CTCs  had  a  different  institutional  focus  (e.g.,  collective  training  for  mid  to  heavy  intensity 
combat  training  at  NTC,  low  intensity  combat  training  at  JRTC),  assessment  and  feedback 
mechanisms  were  generally  comparable  for  all.  Performance  measurement,  based  on  Army 
doctrine,  was  conducted  by  the  OCs  at  all  the  CTCs  and  presented  in  AARs  and  THPs.  Once  these 
CTCs  were  operational,  ARI-POM  investigated  their  data  production  and  formats  to  determine  how 
these  data  could  be  integrated  into  the  CTC  Archive  (Nichols,  1989).  Collection  and  processing 
requirements  were  identified  and  met.  The  composition  and  types  of  the  data  generated  at  the 
CTCs  varied  from  CTC  to  CTC  (See  Table  4.).  The  dynamic  nature  of  the  CTCs  such  as  the 
frequent  changes  to  rules  of  engagement,  training  scenarios,  and  unit  performance  measurements 
required  the  CTC  Archive  be  flexible  and  dynamic  as  well.  While  ARI-POM  staff  sought  to 
maintain  flexibility,  a  tremendous  effort  was  made  to  integrate  the  various  data  into  a  useable 
database.  At  this  point  the  NTC  Archive  became  the  CTC  Archive.  In  response  to  these  additional 
data  sources,  ARI-POM  designed  an  objective  data  collection  system. 

The  volume  of  data  which  began  arriving  at  this  time  was  staggering.  Combined,  there  were 
33  rotations  each  year  at  the  CTCs.  One  CTC  rotation  could  produce  3,500  pieces  of  paper  (THPs, 
overlays,  etc.),  200  video  tapes,  10-50  floppy  disks,  10-35  audio  tapes,  and  extensive  digital  data 
(e.g.,  battlefield  replay,  event  logs).  A  system  of  integrating  and  presenting  the  data  was  created. 
The  data  also  arrived  in  various  formats  such  as:  computer  data  tapes  containing  instrumented  data 
from  the  NTC,  versus  diskettes  from  the  JRTC  containing  I-MILES  data;  text  data  on  diskette, 
primarily  THPs  from  the  NTC  and  JRTC,  versus  hard  copy  THPs  from  the  CMTC;  various  paper 
data  from  all  the  CTCs  (e.g.,  maps,  overlays,  charts);  AAR  video  tapes  were  sent  from  all  CTCs; 
40-track  audio  tapes  from  the  NTC  and  CMTC;  and,  T&EO  data  from  the  JRTC  in  SPSS-PC 
format  on  diskette.  Of  these  data  sources,  Nichols  (1990)  reported  that  the  JRTC  submitted  T«&EO 
hard  copy  data,  not  digital  data.  While  the  CMTC  collected  T&EO  data,  it  did  not  systematically 
send  this  data  to  the  archive.  The  primary  data  sources  used  were  the  replay  and  mission  databases, 
followed  by  the  paper  copy  of  THPs.  Graphics  and  AAR  videos  were  accessed  to  a  much  lesser 
degree  (5%  of  the  studies  conducted  up  to  1990). 
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Table  4.  CTC  Data  Sources  (Adapted  from  Lewman,  et  al.,  1992). 


Sources  of  CTC  Data 

NTC 

JRTC 

CMTC 

Take  Home  Packages 

/ 

/ 

/ 

After  Action  Reviews 

/ 

/ 

/ 

Operations  Orders/Plans 

/ 

/ 

/ 

Map  Overlays 

/ 

/ 

/ 

Operation  Logs 

/ 

/ 

/ 

Observer  Controller  Logs 

/ 

/ 

Communications  Tapes 

/ 

/ 

T&EO  Checklists 

/ 

/ 

Firing  Events 

/ 

/ 

Casualty  Information 

/ 

y 

/ 

Player  Position  Location 

/ 

PlayerA^ehicleAVeapons  Status 

/ 

During  this  time  (1989-90)  the  NTC  upgraded  the  instrumentation  capability  to  track  more 
players  on  the  battlefield.  The  new  system  could  track  up  to  2,000  players  as  opposed  to  the 
previous  500.  The  software  was  also  rewritten  which  compelled  ARI-POM  to  rewrite  all  the 
software  to  access  the  instrumented  data.  The  resulting  database  was  much  more  comprehensive 
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and  reliable  (Baldwin,  1995^). 

During  the  next  five  years,  ARI-POM  developed  and  implemented  a  Training  Research 
Automated  Catalog  System  (TRACS)  which  listed  CTC  data  sources  in  a  single  database  and 
provided  both  an  operator  and  user  interface  capability  (Lewman,  et.  al,  1989).  TRACS  simplified 
the  analyst's  work  to  determine  what  data  were  available  for  a  particular  study.  TRACS  provided 
summary  data,  a  catalog  of  data  sources,  and  data  locations.  From  this  information  an  analyst  could 
automatically  determine  which  rotations  had  the  data  of  interest  and  access  the  improved  CTC 
digital  databases  (Briscoe,  1990). 

TRACS  was  a  menu-driven  catalog  of  the  data  available  in  the  CTC  Archive.  To  use 
TRACS,  the  user  determined  a  research  issue  to  pursue  and  then  determined  the  selection  criteria 
to  support  the  research.  The  menu-driven  TRACS  system  guided  the  user  through  a  series  of  steps 
by  which  to  sample  missions  for  analysis.  For  example,  if  an  analyst  was  interested  in  how  units 
were  performing  during  a  counter  attack  mission  since  the  addition  of  a  new  weapon  system,  the 
analyst  would  select  the  mission  type  (e.g.,  counter  attack)  and  rotations  (e.g.,  87-07)  which 
followed  the  addition  of  the  new  weapon  system. 

Several  programs  were  created  to  facilitate  analysis  of  the  data  and  provide  analysts  tools 
with  which  to  access  the  types  of  information  required  (See  Table  5).  These  tools  were  available 
through  the  use  of  PCS  which  meant  analysts  could  access  and  process  the  data  from  remote 
locations. 

Table  5.  Programs  &  Tools  for  PCs. 


Task  Force  Battle  Trace  (TBAT)  Provided  snapshots  of  the  battle  flow. 

The  Mission  Performance  Analysis  Replay  Tool  (MPART)  Graphically  portrayed  NTC 
battles  in  a  real-time  event  flow. 

General  Purpose  NTC  Analysis  of  Training  Tool  (GNATT II).  Permitted  analysts  to  replay 
NTC  training  missions  on  any  MS-DOS  computer. 

Graphics  Database  Graphics  previously  only  available  in  paper  copy. 


The  Combat  Operations  Research  Facility  (CORF)  was  conceived  and  implemented  at  ARI- 
POM  to  facilitate  research  by  both  military  and  research  analysts.  The  CORF  included  digital  data 
and  hard-copy  products.  Later,  in  1991,  a  Battle  Damage  Assessment  Database  (BDA)  which 
contained  BDA  statistics  on  a  mission  by  mission  basis  was  added.  The  BDA  database  was 
derived  from  the  THPs. 

By  October  of  1990,  ARI-POM  had  developed  the  Combat  Analysts'  Workstation  (CAW). 


’  Personal  communication  September  1995. 
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This  was  a  digital  system  that  enabled  analysts  to  access  data  from  a  personal  computer.  The  CAW 
was  created  based  on  a  government  furnished  Intelligent  Gateway  Processor  (IGP)  software, 
developed  by  Lawrence  Livermore  Laboratory.  The  mechanism  used  to  integrate  and  facilitate  the 
use  of  the  available  databases  was  the  Combat  Training  Data  Gateway  (CTDG)  developed  by  ARI- 
POM  (Briscoe,  1991).  CTDG  was  a  PC-based  bulletin  board  system  which  provided  remote 
access  to  TRACS,  other  VAX-based  databases,  mission  replay  files,  and  graphics  files  (Briscoe, 
1991).  As  TRACS  developed  it  became  known  as  the  Automated  Finder's  Guide  (Mulshine  & 
Murphy,  1994).  A  guide  to  train  and  assist  analysts  in  the  operation  of  the  CAW  and  CORF  was 
written  and  produced  (Nichols  &  Shillcock,  1990).  The  CAW  was  designed  to  allow  Army 
analysts  remote  access  from  anywhere  in  the  world  to  a  variety  of  CTC  data. 

Training  on  the  archive  tools  was  the  principal  means  for  making  the  data  available  to  Army 
personnel.  The  CAW  was  a  user-friendly  system,  but  required  training  and  certification  for 
purposes  of  accessing  the  data.  In  December  1992  ARJ  expanded  the  workshop  to  two  weeks  to 
cover  all  archive  tools  and  the  research  process.  To  support  this  expansion,  CAC  developed  an 
extensive  research  facility  at  POM  that  included  16  CAWs.  The  workshop  was  designed  to  have 
two  phases:  a)  development  of  skills  to  extract  data  relevant  to  an  issue;  and,  b)  conduct  of  research 
and  preparation  of  a  briefing  on  the  issue. 

The  phase  on  development  of  skills  followed  the  Mager  Criterion-Referenced  Training 
(CRT)  Model.  In  accordance  with  that  model,  training  proceeded  by  module.  Each  module 
included  a  skills  assessment,  a  series  of  practical  exercises  with  text  to  prepare  participants  for  the 
skills  assessment,  and  feedback  on  the  skills  assessment.  Participants  worked  through  the  modules 
at  their  own  pace,  assisted  by  facilitators  from  ARI  and  supporting  contractors.  The  module  on 
preparing  a  research  plan  culminated  with  a  small  group  meeting  where  each  participant  described 
the  plarmed  research  approach. 

The  bulk  of  the  two  weeks  was  devoted  to  independent  research  on  the  issues.  ARI  and 
contractors  assisted  with  detailed  data  requests  and  data  analysis.  At  the  end  of  the  second  week, 
each  participant  presented  the  results  of  his  or  her  research  in  a  briefing.  The  briefing  was  the  basis 
for  certification  to  use  the  archive.  The  reports  that  were  suitable  for  CALL  to  certify  the  researcher 
are  included  in  Table  6. 

A  1993  briefing  by  CALL  (Wolff)  identified  35  studies  and  products  within  TRADOC  to 
illustrate  the  value  of  the  CTC  Archive.  In  addition,  28  studies  were  completed  during  the  two- 
week  CTC  Archive  Training  Course.  The  63  resulting  studies  and  products  are  summarized  by 
DTOML  in  Table  7.  The  agencies  labeled  "Other"  in  that  table  include  government  agencies  and 
civilian  research  corporations. 
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Table  6.  Issues  by  DTOML  for  TRADOC  Schools  and  Centers  and  Other  Organizations. 


DTOML 

School  or 
Center 

Issue 

Doctrine 

CAC 

Logistics  on  the  battlefield 

Employment  of  recon  units  in  defense*'’ 

Armor 

Battalion  level  attacks  on  two  axes 

Hea-vy/Light  employment  in  the  desert 

Infantry 

Light  infantry  doctrine  &  training 

Aviation 

AH64  &  OHS  8  employment  on  the  battlefield 

Engineer 

Combined  arms  breaching  operations 

Brigade  level  obstacle  synchronization 

Intel 

GSR  employment 

MI  and  company  team  operations 

Quality  of  R&S/counter-recon  plan* 

Contributions  of  lEW  activities* 

CASCOM 

Effectiveness  of  BSA  defense 

MP 

Use  of  MPs  in  assessing  BCC  and  rear  operations 

Chemical 

Use  of  NBC  and  its  impact  on  the  mission 

Chaplain 

Factors  that  contribute  to  Bde  Ministry  Team 
effectiveness* 

Doctrinal  training  and  employment  of  chaplains 

Effectiveness  of  Bn  TF  CTCP* 

Other 

Impact  of  smoke  and  obscurants  on  M/CM/S  BOS* 

■'Asterisk  indicates  study  was  developed  during  two-week  CTC  Archive  Training  Course. 
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Table  6.  Cont'd 


DTOML 

School  or 
Center 

Issue 

Training 

CAC 

Breaching  operations 

Reconnaissance 

RC  brigades  during  preparation  for  Desert  Storm* 

C&GS  POI  update  (AAR) 

Armor 

Upgrade  of  POI 

Infantry 

Mortars  and  antiarmor  at  JRTC 

Artillery 

Intel 

Firing  rates  and  effects  at  the  NTC 

Integration  of  mortars* 

Reconnaissance  and  surveillance 

Upgrade  of  POI 

AHS 

Training  trends  of  medical  staff  at  brigade  and  battalion 
Medical  training  problems 

MP 

Trends  in  MP  platoon  operations* 

Other 

AD  early  warning  effectiveness* 

Impact  of  early  NTC  missions  on  late  missions  for  RC 
and  AC  units* 

Organization 

CAC 

Artillery  organization  (battery  vs.  platoon)  pre-  and 
post-Gulf  War* 

Armor 

Scout  force  composition 

JFK  Spec. 
Warfare 

Link-up  between  special  operation  forces  and  maneuver 
elements* 

Materiel 

Armor 

Vulnerability  of  scout  vehicles:  APC,  BFV,  and 
HMMWV* 

Infantry 

IFV  requirements  and  performance  at  NTC 

ITV  and  TOW  performance 

Lethality  and  survivability  of  E  Co.  M2  and  ITV* 

Intel 

Automation  of  brigade  S2  intel  preparation  using 
sensors  at  EAB* 

CASCOM 

Refueling  on  the  move 

Logistics  automation  use  and  requisition  flow 
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Table  6  Cont'd 


Materiel  con'td 

Signal 

Radio  net  benchmarking 

Communications  procedures  and  equipment* 

Other 

Comparison  of  M60A3-M1 13  units  and  Ml-Ml  13 
units* 

Leadership 

CAC 

Casualty  evacuation 

Fratricide 

Relation  between  brigade  rehearsal  techniques  and 
synchronization* 

Impact  of  DST  in  defense* 

Armor 

Impact  of  DST  in  offense* 

Infantry 

Staff  force  planning  and  integration 

Engineer 

Impact  of  rehearsals  on  performance 

Artillery 

ID  problems  of  FSO* 

Intel 

Staff  planning  and  reporting 

AHS 

Casualty  rates 

Causes  of  died-of-wounds  casualties 

Other 

Impact  of  command  and  control* 

Fratricide  rates  by  type  of  mission,  weapon,  and  time  of 
day  for  Armor* 

RC  fire  support  planning* 

Fratricide  rate:  OPFOR  and  BLUEFOR* 

CAS  integration* 

In  June  1994  the  scope  of  the  workshop  was  reduced  to  a  one-week  course  on  skills 
development  to  extract  data.  During  that  workshop,  participants  were  also  trained  in  procedures  to 
access  data  through  a  modem  connection.  The  concept  was  that  they  would  complete  their  research 
using  the  remote  procedures  at  their  home  installation  with  certification  based  on  the  reported 
results  to  CALL.  The  certified  participants  would  then  conduct  the  workshop  at  their  location  using 
training  materials  focussed  on  extracting  data  via  modem. 

The  materials  to  support  the  remote  workshop  were  finalized  after  the  June  1994  on-site 
workshop.  The  ARI  monitor  (Hamza)  validated  those  materials  through  a  workshop  at  the  U.S. 
Military  Academy,  West  Point,  NY.  No  other  organization  conducted  a  remote  workshop. 

The  number  of  people  who  received  training  on  the  archive  at  POM  is  summarized  in  Table 
7.  Since  access  to  the  data  was  contingent  on  completing  the  workshop,  the  summary  of 
participants  is  a  rough  indicator  of  interest  in  the  CTC  Archive  data. 
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About  two-thirds  of  the  participants  came  from  TRADOC.  Such  a  distribution  is 
understandable  since  CAC  was  proponent  for  the  archive  in  general  and  the  workshop  in  particular. 
The  type  of  agency  most  likely  to  seek  access  to  the  archive  was  a  TRADOC  school  or  center, 
spurred  largely  by  General  Thurman's  directive.  Seventeen  schools  or  centers  sent  members  of  their 
faculty  and  staff  to  be  trained  at  the  archive  due  to  General  Thurman’s  directive. 

Among  government  research  development  test  and  evaluation  agencies,  the  Army  Research 
Institute  had  the  highest  number  of  people  attending  the  workshop.  "Other"  government  agencies 
included  Operational  Test  and  Evaluation  Center  (Army  and  Air  Force),  Livermore  Laboratories, 
Diamond  Laboratories,  Concepts  Analysis  Agency,  Defense  Institute  for  Training  Resources 
Analyses,  and  Defense  Advanced  Research  Projects  Agency.  Fourteen  civilian  corporations  sent 
staff  members  to  the  archive  training.  Corporations  with  more  than  one  participant  were  BDM, 
PRC,  HumRRO,  Rand,  MPRI,  and  Litton.  Participants  from  FORSCOM  came  from  FORSCOM 
HQ,  7th  ID(L),  and  a  CTC  engineer  company.  The  "Other"  organization  types  include  USAREUR 
and  7th  Army,  Special  Operations  Command,  National  Security  Agency,  Information  Systems 
Command,  U.S.  Military  Academy,  and  the  National  Guard  Bureau. 

Table  7.  Summary  of  Workshop  Participants. 


Type  of  Organization 

Initial  Workshop 

CRT 

Workshop 

Total 

TRADOC 

Schools 

184 

23 

323 

CAC 

43 

6 

TRAC 

30 

3 

CTC 

27 

2 

HQ 

5 

0 

Government  RDT&E 

ARI 

42 

7 

76 

Naval  Post-Grad  School 

17 

0 

Other 

8 

2 

Civilian  R&D 

29 

11 

40 

Army  Materiel  Command 

5 

4 

9 

FORSCOM 

7 

0 

7 

Corp  of  Engineers 

6 

0 

6 

Other 

12 

5 

17 

Total 

415 

63 

478 
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CHAPTER V 

DATA  SOURCES  AND  ARCHIVE  TOOLS 

The  following  is  a  description  of  the  various  data  sources,  evolution  of  the  data,  and  the 
data  processing  undertaken.  In  the  beginning  stages,  since  the  data  sources  were  not  collected  for 
the  purposes  of  analysis  but  were  products  generated  for  training,  the  data  were  examined  for  utility 
and  accessibility  to  researchers.  Although  the  data  sources  were  perhaps  incomplete,  early  studies 
revealed  the  imprecedented  resource  these  data  potentially  provided  to  the  Army. 

Early  data  sources  primarily  consisted  of  data  generated  by  the  instrumentation  system  and 
data  collected  by  human  observers  (i.e.,  OCs)  (Banks,  1993).  The  data  sources  primarily  accessed 
by  analysts  were  the  digital  data  and  the  THPs.  The  data  sources  had  not  changed  dramatically 
until  1995  when  the  THPs  were  extensively  streamlined.  Additional  types  of  data  were  received 
with  the  implementation  of  JRTC  and  CMTC.  The  capability  to  ^alyze  and  obtain  meaningful 
information  from  the  data  changed  substantially  from  1985  to  1995.  The  following  is  a  review  of 
the  data  sources  sent  to  ARJ-POM  and  the  evolutions  they  underwent  (Baldwin  1994;  Briscoe  & 
Kemper,  1986,  Kemper,  1986;  Ford  &  Hamza  1992;  Hutton  1994;  Keesling,  1995;  Nichols,1990; 
Whitmarsh,  1987;  Zimmerman,  Nichols,  &  Kemper,  1986). 

Instrumented  Data 


Although  initially  fraught  with  difficulty  and  breakdowns,  the  digital  information  from  the 
NTC  would  become  the  predominant  data  source  utilized  at  the  CTC  Archive.  The  data  were  not 
available  for  use  until  substantial  programming  and  compiling  had  been  completed.  By  the  time  the 
CTC  Archive  was  transferred  to  Ft.  Leavenworth,  it  included  a  replay  program  of  battlefield  events, 
a  computerized  catalogue  of  all  data  available  at  the  CTC  Archive,  CTC  performance  information, 
automated  THPs,  and  statistical  information  derived  from  the  CTCs'  instrumentation  systems. 

The  NTC  Instrumented  System  (NTC-IS)  consisted  of  four  subsystems:  the  Range  Data 
Measurement  Subsystem  (RDMS),  Range  Monitoring  and  Control  Subsystem  (RMCS),  Core 
Instrumentation  Subsystem  (CIS),  and  the  Live  Fire  Subsystem  (LFS).  The  RDMS  and  CIS 
contained  the  primary  digital  data  collected  at  the  NTC  (Briscoe,  1986;  Whitmarsh,  1987). 

The  RDMS  collected  the  raw  field  data  pertaining  to  'real-time'  position/location  and 
engagement  event  data  on  all  players  equipped  with  telemetry  instrumentation  during  training  with 
the  Multiple  Integrated  Laser  Engagement  System  (MILES).  Real-time  data  is  characterized  by 
being  collected  immediately  as  the  event  occurs.  The  CIS  provided  all  'real-time'  data  processing 
and  interactive  play  capabilities  needed  to  monitor,  command,  and  control  all  engagement 
simulations  which  was  manually  input  into  the  NTC-IS.  The  RMCS  monitored  and  controlled  all 
activities  on  engagement  simulations  and  derived  data  from  the  RDMS  and  CIS  systems  (Briscoe, 
1986;  Whitmarsh,  1987).  The  LFS  derived  data  from  the  CIS  and  RDMS  subsystems  (Briscoe, 
1986). 
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The  NTC-IS  Data  Tape  contained  battle  replay  information  for  display  on  NTC  workstation 
graphic  terminals.  This  data  was  termed  NTC  History  Data.  The  data  in  these  files  were  generated 
from  the  field  instrumentation  system  and  operator-entered  interactive  command  and  control  inputs. 
The  field  instrumentated  data  were  captured  via  transmission  from  digital  posts  at  various  points  on 
the  battlefield.  The  interactive  command  and  control  inputs  were  conducted  at  the  TAP  where  OCs 
could  call  in  information  and  "kill"  players.  Input  of  indirect  fire  simulated  kills  was  also 
conducted  at  the  TAP  (SAI,  1984). 

ARI-POM  conducted  an  extensive  study  to  examine  the  NTC  data  collection  systems  and 
the  existing  database  to  ascertain  accessibility  and  utility  for  ARI  and  Army  research.  This  study 
investigated  the  format  and  character  of  the  NTC  data  which  resulted  in  a  detailed  documentation 
of  the  two  NTC  digital  data  sources,  the  log  tapes  from  the  RDMS  and  back  up  files  from  the  CIS 
(Pobes,  1984). 

Instrumented  raw  data  consisted  of  information  on  player  position,  direct  fire  weapons 
events,  and  communications.  Player  position  measurements  were  continuously  updated  every  60 
seconds,  except  for  those  "killed".  Weapon  events  for  direct  fire  weapon  simulators  were  divided 
into  two  categories:  a)  a  firing  event  when  a  weapon  was  fired,  and  b)  a  kill.  Kills  were  "paired" 
with  weapon  effect  events  to  assign  a  target  to  a  weapon  for  computing  player  performance 
statistics.  Indirect  fire  kills  and  minefield  event  kills  were  OC  generated  and  manually  input  into 
the  database.  Communications  were  recorded  for  every  depression-release  conducted  by  a  radio 
operator.  The  transmission  time  and  number  of  transmissions  made  were  computed. 

The  NTC-IS  archived  player  position,  weapon  and  communications  event  data,  OC 
observations,  alert  messages  (e.g.,  controller  gun  events,  indirect-fire  firings,  weapon-target 
pairings)  and  free  format  manually  input  messages.  These  data  were  stored  in  a  Player  History  File 
(PHF)  for  each  exercise  segment.  Segments  were  discrete  portions  of  the  overall  Electronically 
Simulated  (ES)  or  Live  Fire  (LF)  exercise  and  delineated  by  natural  breaks  in  the  engagement,  a 
transition  between  tactical  missions,  movement  to  a  new  terrain  area,  a  major  change  in 
environmental  conditions,  or  by  a  command  decision.  These  segments  lasted  up  to  48  hours,  but 
generally  lasted  8  to  12  hours  (Fobes,  1984). 

The  RDMS  provided  real  time  player  position  location  and  engagement  event  data  on  all 
instrumented  players. 

The  RMCS  provided  the  means  to  monitor  and  control  activity  on  the  ES  and  LF  ranges. 
The  capabilities  included  automated  and  human  sensors  and  a  communications  component  to 
connect  these  sensors  together  and  with  the  CIS.  The  CIS  provided  real  time  data  processing  and 
display  capabilities  required  to  monitor,  command  and  control  the  Engagement  Simulation  (ES)  and 
Live  Fire  (LF)  exercise  activities.  The  CIS  also  provided  the  data  processing  capabilities  required 
to  synthesize  and  present  near  real-time  AAR  and  THPs.  CIS  also  provided  the  data  processing  and 
display  capabilities  required  to  support  Training  Developments  and  Combat  Developments  research 
with  NTC  data. 
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The  CIS  Log  consisted  of  initialization  data,  player/unit  update  information,  RDMS  event 
messages,  firer-target  pairings,  position/location  (PL),  control  measures,  and  indirect  fire  casualty 
assessments  (IFCAS).  This  data  usually  arrived  on  5-10  magnetic  tapes  per  task  force  per  rotation, 
contained  15-30  mission  segments  per  task  force,  4  global  files  per  task  force,  and  9-12  mission 
specific  files.  CIS  log  provided  data  for  the  NTC  Database  Research  System  (NTCDRS)  and  the 
NTC  Workstations.  Unlike  all  other  data  sources,  this  information  was  available  for  all  NTC 
rotations.  This  important  factor  prompted  to  researchers  to  examine  CIS  Logs  very  closely. 

The  NTCDRS  was  a  system  developed  by  SAI  to  convert  the  log  tapes  generated  on  the  CIS 
into  a  form  which  would  facilitate  analysis  (SAI,  Final  Report,  1984).  The  taped  data  were 
formatted  for  the  INGRES  Relational  data  base  management  system  (RDBMS)  on  the  DEC-VAX 
1 1/780  computing  system.  INGRES  was  selected  based  on  the  research  database  (RDB)  criteria 
which  were  to  maximize  data  element  correlations,  be  user  friendly,  be  retrieval  efficient  and 
modularly  expandable.  It  was  also  to  allow  researchers  reliable  information  and  meet  computer 
system  specifications.  This  system  was  implemented  as  a  series  of  computer  programs  which 
produced  a  series  of  relational  database  tables.  These  tables  were  to  be  used  to  derive  reports  and 
rudimentary  statistics.  INGRES  provided  information  for  research  by  manipulating  the  existing  61 
data  tables. 

However,  there  were  major  flaws  with  this  software  program;  the  primary  one  being  that 
only  one  mission  at  a  time  could  be  examined  or  retrieved,  hence  there  was  no  capability  to  build 
a  large  enough  database  to  actually  derive  anything  more  than  immediate  data. 

The  RDMS  Log  contained  all  data  passed  from  the  RDMS  to  the  CIS  such  as  field  events 
including  MILES,  player  position/location,  and  status  messages  for  the  RDMS  components.  While 
some  of  the  raw  data  were  systematically  transferred  to  the  CIS,  other  data  remained  solely  on  the 
RDMS  log  tapes,  such  as  the  result  (HIT,  NEAR  MISS,  or  KILL)  paired  with  weapon  type.  This 
information  was  input  from  the  the  Position  Tracking  Computational  Component  (PTCC)  of  the 
RDMS  and  required  CIS  initialization  information  to  be  useful.  RDMS  Logs  occupied  a  series  of 
15-30  magnetic  tapes  per  rotation.  Though  a  valuable  source  of  information  and  an  important 
contributor  to  the  NTCDRS,  the  RDMS  Log  data  was  not  organized  by  task  force  or  mission  and 
was  not  available  for  all  NTC  rotations  (Briscoe,  1986). 

The  RDMS  tapes  were  compared  to  the  CIS  tapes  and  a  strong  correlation  was  found 
(Briscoe,  1 986).  Software  was  then  developed  to  process  the  RDMS  log  data  and  to  incorporate 
this  data  into  the  redesigned  database,  thus  supplementing  the  CIS  data. 

Once  the  instrumented  data  began  arriving  at  ARI-POM  an  examination  was  made 
regarding  the  reliability  of  the  data.  The  data  were  indeed  skewed  due  to  instrumentation 
difficulties,  such  as  noninstrumented  vehicles,  weapons,  and  personnel;  equipment  failure;  terrain 
masking;  pairing  errors  of  killer  weapon  to  vehicle  killed;  and  software  problems.  Additionally,  the 
capability  to  support  systematic  analyses  was  limited  because  of  the  following  reasons  (Banks  & 
Whitmarsh,  1987;  Briscoe,  1986;  Doherty,  1987): 
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1.  The  programming  was  such  that  only  one  set  of  data,  one  mission,  could  be  retrieved. 
The  software  'hard-coded'  the  resulting  database  named  as  ARIDATA.  The  result  was 
that  each  time  a  mission  was  retrieved,  the  previous  mission  database  was  erased  or 
written  over.  This  precluded  any  ability  to  build  a  database  from  which  to  derive  long 
term  or  even  rotational  lessons  learned. 

2.  The  software  system  was  prone  to  catastrophic  failure  at  several  junctions. 

3.  The  system  was  operated  through  a  number  of  command  files  which  were  badly 
documented,  awkward  to  use,  and  error-producing. 

4.  The  NTCDRS  databases  could  support  only  a  small  percentage  of  the  research  efforts 
because  the  database  was  limited  to  the  CIS  digital  data. 

5.  The  data  within  the  INGRES  tables  of  NTCDRS  were  difficult  to  use,  the  queries 
required  extensive  time. 

6.  The  NTCDRS  database  was  unreliable  because  it  contained  multiple  occurrences  of 
identical  data,  and  difficult  to  use  data  relations  and  keys. 

ARI-POM  then  set  about  designing  a  database  which  would  contain  not  only  the  digital  data 
in  an  easy  to  use  format,  but  also  the  hard-copy  data  needed  to  derive  a  complete  picture  of  NTC 
rotations.  The  GAO  letter  of  1986  spoke  of  the  need  to  design  a  database  which  facilitated  research 
and  was  based  on  research  requirements.  Not  since  the  inception  of  the  NTC  had  any  effort  been 
made  to  address  the  needs  of  the  research  community.  In  response  to  these  concerns,  the  redesign 
included  the  following; 

1.  Requirements  Analysis.  While  the  potential  use  of  these  data  sources  was  great,  the 
realization  of  this  potential  was  deterred  by  the  lack  of  information  provided  to  the 
military  and  scientific  community  on  what  kind  of  data  were  available  or  how  to 
properly  interpret  the  data  with  all  their  limitations.  Hence,  ARI-POM  solicited 
information  from  potential  NTC  data  users  about  what  they  required.  The  Infantry 
School  at  Fort  Benning,  the  Armor  School  at  Fort  Knox,  and  the  CAC  at  Fort 
Leavenworth  were  contacted.  Requirements  such  as  data  types,  definitions,  quality, 
units,  etc.  were  included  in  this  step.  This  effort  was  responsive  to  the  GAO  (1986) 
finding  that  the  research  community  needed  to  be  contacted  for  their  input  regarding 
data  needs. 

2.  Database  Structure  Design.  The  table  structures,  data  relations,  and  keys  were  defined. 
Complete  specifications  and  documentation  were  made  of  all  data  sources,  including 
format  information  and  processes  needed  to  incorporate  the  data  into  the  database. 

3.  Processes.  This  phase  concerned  the  development  of  processes  by  which  to  build  the 
new  databases.  The  processes  included  computer  programs,  command  files,  manual 
entry,  data  verification  and  the  derivation  of  some  data  elements. 

4.  Implementation.  All  the  separate  entities  developed  in  the  previous  steps  were 
integrated  and  the  NTCDRS  format  was  retrofitted. 

ARI-POM,  undertook  to  correct  the  instrumented  data  problems  with  assistance  from  BDM. 
By  September  of  1985,  the  NTCDRS  was  operational.  The  database  redesign  was  initiated  by 
defining  an  approach  culminating  in  a  database  system  that  supported  the  requirements  of  the  Army 
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and  ARI.  Next,  a  preliminary  design  was  published  and  distributed  to  all  potential  users  of  the 
NTC  data  identified  at  that  time.  Comments  and  suggestions  were  considered  and  incorporated 
through  a  modified  Delphi  technique  into  the  final  revision  of  the  basic  design,  and  development 
commenced.  The  database  building  software  was  completed  in  May,  1987,  at  which  time  mission 
databases  were  developed. 

A  more  complete  database  was  accomplished  by  writing  a  VAX  FORTRAN  program 
General  Dynamics  Electronics  Tape  (GDETAP)  to  read  the  GDE  log  tapes  which  contained  the 
RDMS  log  information  produced  at  the  NTC.  This  program  and  use  of  this  log  was  undertaken 
because  the  RDMS  logs  contained  all  the  information  passed  to  the  CIS  (NTCDRS),  plus  additional 
information  which  was  not  passed  (Briscoe,  1986  (a)).  After  the  completion  of  this  project, 
approximately  30  utility  programs  and  command  files  were  developed  to  enable  researchers  to 
analyze  the  data.  The  systems  needed  by  the  analysts  included  reloading  specific  tables, 
documenting  database  contents,  and  dumping  intermediate  files.  More  than  500  mission  segments 
were  quickly  processed  by  NTCDRS,  which  contained  both  the  CIS  and  RDMS  information,  and 
were  made  available  to  researchers.  This  was  the  first  time  the  digital  data  generated  at  the  NTC 
provided  information  for  researchers.  Several  studies  were  conducted  using  the  improved  database 
and  useful  information  provided  to  the  Army.^  The  RDMS  log  information  also  provided,  once 
processed  through  the  GDETAP  program,  a  check  and  validation  of  the  NTCDRS.  The  comparison 
of  data  yielded  consistency  between  the  two  sources  (Briscoe,  1986  (b)). 

By  1988,  ARI  had  identified  19  tables  from  the  mission  databases  and  had  developed 
programs  to  extract  the  tables  (McCluskey  &  Lewman,  1988)  (See  Table  8.).  Three-himdred 
missions  of  data  were  loaded  and  available  for  research.  Although  awkward  to  manipulate,  a 
system  had  been  developed  and  the  ability  to  conduct  trend  analysis  was  available.  Programs  with 
which  to  derive  and  build  databases  were  developed  and  user  manuals  written  (Baldwin,  1990). 


*See  Appendix  A  for  a  summary  of  studies  conducted  during  this  time  period. 


28 


CTC  Archive 


Table  8.  1988  Mission  Databases. 


Mission  Databases  Available  by  1988 


Mission  Identification  Table  (MD) 

Player  State  Initialization  Table  (PSIT) 

Player  State  Update  Table  (PSUT) 

Unit  State  Initialization  Table  (USIT) 

Unit  State  Update  Table  (USUT) 

Unit  Type  Table  (UTT) 

PlayerA^ehicle/Weapon  Code  Table  (PVWT) 
Firing  Event  Table  (FET) 

Pairing  Event  Table  (PET) 

Communication  Table  (CT)  - 

Ground  Player  Position  Location  Table  (GPLT) 
Air  Player  Position  Location  Table  (APLT) 
IFCAS  Target  Table  (IFTT) 

IFCAS  Target  Group  Table  (IFGT) 

IFCAS  Missions  Fired  Table  (IFMF) 

IFCAS  Casualties  Table  (IFCT) 

Minefield  Casualties  Table  (MCT) 

Control  Measure  Table  (CMT) 

Control  Measure  Add  Table  (CMA) 


Although  the  digital  data  had  been  deciphered  ^and  processed,  Lewman's  (1988)  study 
indicated  a  redesign  of  the  database  was  required  for  several  reasons,  they  were: 

1 .  The  original  design  was  inefficient  because  a  great  deal  of  redundant  information  was 
present  in  several  tables; 

2.  This  redundancy  caused  the  databases  to  be  inordinately  wasteful  of  disk  space,  limiting 
the  number  of  databases  that  could  be  maintained  online  at  any  particular  time;  and 

3.  The  process  required  to  build  the  databases  was  unwieldy,  yet  provided  little  or  no 
opportunity  to  control  the  scope  of  the  databases  that  were  built. 

Combat  Analyst  Workstation 

The  redesign  included  a  digital  system  for  analysts,  termed  the  Combat  Analysts' 
Workstation  (CAW)  (Nichols,  1990).  This  system  provided  access  to  the  digital  CTC  Archive 
through  a  series  of  programs  which  were  developed,  tested,  and  implemented.  The  CAW  was 
accessible  from  remote  sites  via  a  modem.  The  mechanism  used  to  integrate  and  facilitate  the  use 
of  the  available  databases  was  based  on  the  Intelligent  Gateway  Processor  (IGP)  software, 
developed  by  Lawrence  Livermore  Laboratory.  The  Combat  Training  Data  Gateway  (CTDG)  was 
adapted  from  the  IGP  (Rigg,  1 990).  The  IGP  was  selected  because  it  allows  rapid,  efficient,  and 
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effective  access  to  information  without  disturbing  existing  systems.  CTDG  was  a  PC-based 
bulletin  board  system  which  provided  remote  access  to  VAX-based  databases,  mission  replay  files, 
and  graphics  files  (Briscoe,  1991;  Rigg,  1990).  CAC  provided  14  of  these  workstations  at  ARI- 
POM.  An  interface  system  was  developed  and  implemented  termed  the  Training  Research 
Automated  Cataloging  System  (TRACS).  TRACS  provided  users  information  regarding  all  data 
sources  and  access  to  the  digital  data  processed  at  ARI-POM  and  a  database  located  at  Ft. 
Leavenworth,  the  Army  Lessons  Learned  Management  Information  System  (ALLMIS).  ALLMIS 
provided  textual  lessons  learned  information  and  keyword  selection  capability  (Briscoe,  1990; 
Nichols  &  Shillcock,  1991). 

During  this  stage  the  Mission  Databases  integrated  digital  data  from  the  JRTC  IMILES 
system,  player/unit  descriptions,  player  position  and  events,  definition  of  control  measures  by  type 
and  location,  and  indirect  fire  information.  By  1991,  over  450  NTC  missions  had  been  processed 
into  the  database  along  with  the  capability  upgrade.  A  Mission  Graphics  Database  provided  the 
standard  digitized  images  from  each  mission  (Nichols  &  Shilcock,  1991).  THPs  were  processed 
into  the  digital  database  to  provide  ready  access.  Battle  replay  capabilities  continued  to  be  refined. 

In  1990  a  proposed  upgrade  was  implemented  and  completed  at  the  NTC,  which  had  been 
anticipated  for  2  years.  The  upgrade  entailed  reconfiguring  the  entire  system.  The  "SUN"  was  a 
technological  advance  which  permitted  the  tracking  of  2,000  plus  players.  This  was  an  increase 
from  the  ability  to  track  400  players.  The  consequence,  however,  was  that  ARI-POM  was  required 
to  re-write  all  the  programming  which  had  been  accomplished  in  the  first  phase  of  the  CTC  Archive 
to  retrieve  the  digital  data.  Once  this  was  accomplished  a  new  and  much  more  detailed  database 
emerged.  The  following  tools  were  developed  to  assist  analysts: 

Task  Force  Battle  Trace  ITBATI  TBAT  was  designed  to  provide  snapshots  of  the  battle 
flow.  This  was  accomplished  by  capturing  all  player  movement  during  missions  in  a  single  picture 
presented  with  a  realistic  representation  of  the  terrain.  A  pattern  of  movement  could  be  derived  and 
analyzed  to  determine  maneuver  and  command  and  control  performance. 

The  Mission  Performance  Analysis  Replay  Tool  IMPART!  MPART  was  designed  to 
portray  NTC  battles  in  a  real  time  event  flow  format  graphically  represented  on  a  computer. 
MPART  used  terrain  graphics  with  relief  features  and  grid  lines  for  orientation.  The  analyst  could 
choose  those  individuals  or  collective  groups  of  players  for  display  according  to  need.  Players  were 
displayed  by  symbols  which  represented  their  respective  combat  systems  (tank,  APC,  infantry).  A 
key  feature  of  MPART  was  the  capability  of  using  the  METT-T  templates  to  assist  in  mission 
analysis.  The  METT-T  template  could  be  moved  and  rotated  to  the  proper  position  and  orientation 
during  the  initial  phase.  At  mission  end,  the  number  of  players  in  each  area  of  the  template  were 
automatically  summarized  as  to  kill,  survived,  etc. 

General  Purpose  NTC  Analysis  of  Training  Tool  tGNATT  ID  GNATT II  was  an  evolution 
in  replay  programming  which  permitted  analysts  to  replay  NTC  training  missions  on  any  MS-DOS 
computer  suitably  equipped  with  standard  EGA  color.  In  order  to  use  this  program,  an  analyst 
selected  missions  from  the  NTC  Mission  Databases  on  the  VAX.  The  selected  missions  were  then 
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copied  from  the  VAX  to  diskette  which  were  the  data  for  the  PC  replay  software.  These  replays 
could  then  be  produced  on  the  PC  with  the  PC  software.  Numerous  missions  had  been  derived  by 
ARI-POM,  and  were  available  on  diskette.  GNATT II  was  user-friendly  and  permitted  analysts  to 
do  the  following: 

1.  Display  any  area  of  the  NTC  from  2000  to  less  than  1  square  kilometer. 

2.  Select  any  combination  of  players  by  unit  designation. 

3.  Differentiate  units  and  weapons  systems  by  symbol  type  and  color. 

4.  Display  all  types  of  battlefield  graphics  as  desired. 

5.  Identify  individual  players  using  the  mouse. 

6.  Identify  killed  players. 

7.  Graphically  display  MILES  engagements. 

Graphics  Database  ARI-POM  had  begun  capturing  graphics  teough  video  and  pictures  and 
reading  this  into  the  CTC  digital  database.  The  graphics  listed  below  were  available  through  the 
remote  user's  computer. 

1 .  After  Action  Review  Slides 

2.  Artillery  Mission  Logs 

3.  Artillery  Preps 

4.  Artillery  Mission  Summary  Sheets 

5.  Commander's  Intent 

6.  Decision  Support  Templates 

7.  Event  Templates 

8.  Execution  Matrices 

9.  Firing  Intensity  Profiles 

10.  Fire  Support  Plans 

1 1 .  Situational  Templates 

12.  Obstacle  Reports 

13.  Obstacle  Plans. 

14.  Operations  Orders 

15.  Operations  Plans 

16.  Reconnaissance  Plans 

17.  Reconnaissance  &  Surveillance  Plans 

18.  Scout  Positions. 

19.  Task  Force  Overlays 

20.  Mission  Specific  Graphics  (a  catch-all  for  previously  undefined  graphics  peculiar  to  a 
mission) 

Mission  Database  The  Mission  Database  was  a  relational  database  which  contained  data 
derived  for  the  CIS  at  the  NTC.  After  the  upgrade,  CIS  data  were  collected  on  SUN  Work  Stations 
and  stored,  as  before,  in  real  time  format.  The  Mission  Database  was  a  collection  of  battle/mission 
research  databases  in  which  a  separate  research  database  was  generated  for  each  mission  conducted 
at  the  NTC  during  a  rotation.  The  separate  databases  contained  direct  fire  event  data,  indirect  fire 
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mission  data,  minefield  data,  communications  data,  ground  and  air  player  position  location  data, 
fratricide  data,  and  maneuver  control  measure  data  (Walsh  &  Baldwin,  1995). 

The  year  1994  likewise  saw  another  upgrade  to  the  instrumentation,  but  this  did  not  affect 
the  collection  of  the  digital  information.  There  may  be  more  data  available,  but  at  the  writing  of 
this  report,  this  was  unknown. 

During  the  remainder  of  1994  databases  and  interfaces  were  improved  and  data  routinely 
processed.  At  the  end  of  1994,  the  end  of  the  ARI-POM  participation  Avith  the  CTC  Archive,  the 
following  databases  were  available  ( Baldwin,  Bennett,  Briscoe,  &  Mulshine,  1994;  Mulshine,  1994 
(a);  Mulshine,  1994  (b);  Mulshine,  1994  (c);  Walsh  &  Baldwin,  1995): 

Automated  Finder's  Guide  CAFG!.  AFG  was  an  improvement  to  Training  Research 
Automated  Catalog  System  (TRACS),  but  basically  was  the  same  system.  TRACS  and  the  AFG 
listed  CTC  data  sources  in  a  single  database  and  provided  both  an  operator  and  user  interface 
capability  (Lewman,  et.  al,  1989). 

Automated  THP  Database.  The  Automated  THP  Database  contained  most  of  the  textual 
data  the  CTCs  sent  to  ARI-POM  in  the  THPs.  Files  were  derived  and  organized  by  BOS.  The 
following  rotations  were  available: 

NTC:  Rotations  90-01  to  94-05 

JRTC:  Rotations  90-01  to  94-09 

CMTC:  Rotations  90-01  to  94-02 

Battle  Damage  Assessment  CBDAI  Database.  BDA  Database  contained  force-on-force 
battle  statistics  for  each  mission.  These  data  were  manually  entered  for  NTC  and  CMTC.  JRTC 
sometimes  sent  digitized  BDA,  which  was  then  entered  via  the  computer  system.  Not  all  CTC 
databases  contain  the  same  amount.  The  databases  contained: 

NTC:  Rotations  90-01  through  90-1 1,  90-13,  91-02,  91-10, 91-12, 

CMTC:  Rotations  90-01  through  90-12,  91-01  through  91-10, 

JRTC:  Rotations  90-01  through  90-08,  91-01  through  91-09. 

Take-Home  Packages  (THPl 

THPs,  a  paper  based  data  source,  emerged  early  on  as  a  reliable  data  source.  THPs  were 
developed  for  the  units  to  take  to  home  station  and  improve  their  training.  The  THPs  included  daily 
mission  summaries,  trend  analyses  across  the  seven  Battlefield  Operating  Systems  (BOS),  VAAR, 
and  summary  statistics  on  casualty  figures,  equipment  losses,  and  gunnery  data.  The  THPs  were 
primarily  produced  by  transcribing  OC  comments  and  their  subjective  assessments.  THPs  varied 
widely  in  content  and  formation  because  no  standard  was  enforced  (Chapman,  1992;  Nichols,  1987; 
Whitmarsh,  1987). 
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During  the  early  years,  THP  statistics  such  as  battle  losses,  casualties,  battle  loss  ratios, 
radio  transmissions,  gunnery  and  battle  damage  assessments  were  calculated  by  hand  by  the  OCs. 
Later,  as  the  CIS  became  operational,  these  statistics  were  generated  digitally.  At  the  TAP,  once 
the  CIS  was  operational,  analysts  were  able  to  further  generate  statistics  and  information  for  the 
THPs  and  AARs  (Chapman,  1992;  Shackelford,  1985).  The  information  they  contributed 
consisted  of  the  above  statistics  as  well  as  graphics  generated  by  the  DeAnza  workstations  which 
depicted  critical  times  during  mission  execution. 

The  content  of  the  packages  evolved  from  1982  to  1986  to  include  the  TF  Mission 
Statement/Commander's  concept,  mission  execution  summary,  battle  statistics,  review  by  BOS,  and 
Field  Manual  focused  observations  (Nichols,  1987).  The  THP  was  prepared  for  each  TF 
Commander  by  the  NTC  Training  Analysis  and  Feedback  (TAF)  Division.  The  dociunent 
contained  narrative  and  numerical  descriptions  of  unit  performance  at  the  NTC  and 
recommendations  for  additional  training  at  home  station.  THPs  from  1986  to  1989  included 
(Kemper,  1986): 

•General  Overview  of  the  Purpose,  Scope,  and  Organization  of  the  THP 

•Summary  description  of  the  14-day  training,  including  missions  conducted,  equipment 

losses  on  both  sides  by  mission,  live-fire  performance  measures,  and  performance  trends. 

•After  Action  Reviews  (AARs)  for  each  mission  including: 

-Narrative  description 

-Critical  Events  for  each  Operating  System 

-Statistical  data  on  equipment  tosses  and  radio  transmissions. 

•List  of  videotapes  contained  in  the  THP. 

During  fiscal  1987,  THPs  were  further  standardized  by  issuing  forms  to  OCs  which 
contained  specific  questions  to  be  answered  by  all  OCs  for  each  mission.  The  THPs  contained 
information  on  battle  performance  by  BOS  and  by  Co/Tm.  This  data  collection  system  continued 
until  1995  (Yorther,  1995). 

In  1990,  Engineering  and  Economics  Research,  Inc.  conducted  a  study  of  the  THP  and  its 
utility  at  home  station  for  the  battalion  commander.  While  the  THP  generally  received  approval, 
there  were  many  areas  of  needed  improvement  noted.  One  of  the  more  often  cited  criticisms  was 
that  the  THP  was  received  at  home  station  approximately  3  months  after  a  rotation,  at  which  time 
training  planning  had  already  been  initiated,  much  of  the  original  force  has  been  dispersed,  and  it 
became  history  rather  than  responsive  to  the  training  required.  Generally,  the  THP  was  cited  as  a 
useful  tool,  though  cumbersome  and  disjointed  in  focus  (i.e.,  did  not  focus  on  METL,  but  on  BOS) 
and  provided  no  link  between  the  OCs'  analysis  and  AMTP  training  at  home  station.  The  content 
of  the  THP  at  that  time  was  fairly  consistent  with  the  previous  THPs. 

In  July  1993,  the  General  Officer  Steering  Committee  discussed  the  THPs  being  issued  to 
home  station  from  the  NTC.  They  found  that  THPs  were  not  timely  nor  responsive  to  the  training 
needs  of  the  units.  The  information  contained  in  the  THPs  was  deemed  excessive  and  too  costly. 
On  9  March  1995,  LTC  C.  R.  Hammond,  CTC  Director,  and  Major  S.N.  Welks,  Chief  of  the 
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Operations  Division  of  the  CTCs,  issued  a  FAX  to  all  CTCs  and  other  associated  institutions  with 
instructions  for  the  implementation  of  a  new  format  for  THPs.  The  new  THP  format  was  extremely 
streamlined  and  included  only  the  following:  a)  Commander's  Final  AAR,  b)  Executive  Summary 
and  BOS  Analysis  limited  to  50  pages,  c)  Battle  Summary  Video  Tapes,  d)  maps,  graphics,  and 
charts,  and  e)  Tailored  Information  for  the  Unit  Command  on  request.  No  further  guidance  was 
given.  Each  OC  team  at  each  of  the  CTCs  was  free  to  develop  the  THPs  within  these  restrictions 
(Maj.  Yother,  1995).  The  NTC  decided  to  drop  the  AAR  information  by  Co/Tm  and  began 
including  only  the  TF  by  BOS  analyses  at  the  end  of  the  rotation.  These  new  THPs  were  to  be 
issued  to  the  unit  prior  to  their  departing  the  CTC.  While  greatly  increasing  the  timeliness  of  the 
THPs,  the  utility  to  the  military  research  community  was  greatly  diminished. 

THPs  were  routinely  sent  to  ARI-POM  from  the  NTC,  both  in  hard-copy  and  on  diskette. 
The  JRTC  did  not  use  the  forms  and  sent  much  less  information  (Hutton,  1995;  Yother,  1995). 
However,  since  the  JRTC  was  designated  for  use  by  light  infantry  forces  and  the  NTC  for  heavy 
forces,  it  seems  unlikely  studies  would  be  conducted  across  these  two  CTCs.  The  CMTC  THPs 
were  not  submitted  to  ARI-POM  in  diskette  form,  hence  the  data  extracted  was  labor  intensive 
(Hutton,  1995;  Keesling,  1995®). 

The  early  THPs  from  the  NTC  involved  a  great  deal  of  processing.  They  were  created  by 
compiling  notes  and  remarks  recorded  by  OCs  for  each  mission,  for  each  Co/Tm,  and  processed 
into  BOS  categories.  Maps,  overlays,  and  photos  were  included  when  requested.  The  effort 
required  several  weeks  of  work  to  complete,  and  at  this  time  a  copy  was  sent  to  home  station  and  a 
copy  sent  to  ARI-POM.  THPs  were  a  dense  but  extensive  data  source  for  much  of  the  research 
conducted  through  the  CTC  Archive. 

In  1 989  an  effort  was  undertaken  to  develop  a  DOS-based  version  of  the  THP  database  to 
integrate  the  hard-copy  and  word  processed  versions  of  the  THP  into  the  electronic  database.  In 
1989,  THPs  were  integrated  into  the  digital  database  at  ARI-POM  (Lewman,  et  al.,  1989).  A 
methodology  for  extracting  the  relevant  portions  of  the  THPs  was  developed  and  implemented.  By 
1991,  39  rotations  of  THPs  had  been  loaded  into  the  THP  database  from  the  JRTC  and  the  NTC. 
The  CMTC  had  not  submitted  any  THPs  in  any  form. 

NTC  THPs  typically  cover  six  elements:  Armor  (A)  Task  Force,  Mechanized  (M)  Infantry 
Task  Force,  Forward  Support  Battalion  (FSB),  Aviation  (AVN),  Field  Artillery  (FA)  and  Brigade 
operations  (Bde)  (Ford  &  Hamza,  1992).  They  contained  the  following  sections:  Executive 
Summary/Trends,  Mission  Statements,  Lessons  learned  by  BOS,  Company/Team  Lessons  learned 
by  BOS,  and  Battle  Statistics  (See  Table  9.).  In  the  early  stages,  the  Company/Team  THPs  were 
submitted  as  a  separate  document,  and  were  soon  integrated  into  the  Task  Force  THP. 

The  typical  format  for  the  JRTC  THPs  consisted  of  two  annexes:  Task  Force  Trends,  and 


^Personal  conversation,  September  1995 
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Company/Battery/Platoon  Trends  (See  Table  10.).  Emphasis  was  placed  on  trends  exhibited 
throughout  the  rotation,  systematic  reporting  of  missions  and  battle  statistics  did  not  occur. 
However,  the  JRTC  THPs  were  organized  by  BOS  similar  to  the  NTC  THPs,  and  sometimes 
included  selected  battle  statistics  (Ford  &  Hamza,  1 992). 


Table  9.  NTC  THPs. 


Section 

Contents 

Executive  Summary/Trends 

Overall  performance. 

Trends  and  training  recommendations  by 

BOS 

Mission  Statements 

OC  description 

Direct  quote  fi'om  Commander 

Battlefield  Operating  Systems 

Lessons  learned 

By  BOS 

For  each  mission 

Performance  within  BOS  by  phase  (Plan, 
Prepare,  Execute) 

Lessons  learned  for  each  BOS 

Force  on  force  missions  followed  by  live 
fire. 

NCO  Support  and  Individual  Skills  (if 
included)  after  BOS 

Company/Team  Operating  Systems  and 
Lessons  learned 

By  Company 

Comments  for  each  mission  by  phase 

Lessons  learned. 

Battle  Statistics 

Covers  force  on  force  missions 

Three  parts  per  mission: 

Task  Force  losses  broken  out  by 
•OPFOR  weapons 
•Company/Team  losses 

•Weapon  systems  that  caused  OPFOR 
casualties. 
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Table  10.  JRTC  THPs. 


Section 

Contents 

Task  Force  Trends 

Comments  grouped  by  BOS. 

Typically  included: 

•Critical  Tasks 
•Unit  Strengths 

•Areas  in  Need  of  Improvement 
•Training  Recommendations 

May  include  Mission  Summary 

Battle  statistics  for  selected  units  may  be 
reported  within  BOS 

Company/Battery/Platoon  Trends 

Comments  grouped  by  unit. 

Typically  included: 

•Critical  Tasks. 

•Unit  Strengths. 

•Areas  in  need  of  Improvement. 

•Training  Recommendations 

The  CMTC  produced  very  comprehensive  THPs  in  paper  copy  only.  The  sections  of  the 
CMTC  THPs  consisted  of  a  Mission  Summary,  Mission  AAR  Sheets,  Operations  Orders,  AAR 
Slides,  and  a  Fratricide  Checklist  (See  Table  11.).  Topics  by  BOS  were  systematically  covered  in 
the  Description  of  Performance  portion  of  the  Mission  Summary  section  (Ford  &  Hamza,  1992). 

Table  11.  CMTC  THPs. 


Section 

Contents 

Mission  Summary 

General  Information:  Enemy  situation, 
friendly  situation,  mission  statement,  CDR's 
concept. 

Mission  AAR  Sheets 

Hand-written  OC  observations  by  BOS. 

Operations  Orders 

Includes  FRAGOs  and  graphics. 

AAR  Slides 

Points  emphasized  in  AAR  for  mission. 

Fratricide  Checklist 

Description  of  fratricide  and  contributing 
factors  (e.g.,  weather). 
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Companv/Team  Take-Home  Packages 

The  Company/Team  THPs  were  prepared  by  the  TAP  Division  for  each  TF  company/team 
based  on  instrumented  data  and  input  from  OCs  much  like  the  TF  THPs.  The  document  described 
critical  events  in  the  Plan,  Prepare,  and  Execute  phases  for  each  NTC  mission  for  each 
Company/Team.  These  THPs  were  integrated  into  the  Task  Force  THPs  around  1987  and  were 
only  provided  at  the  NTC.  After  1 994  rotation  9,  company/teams  were  no  longer  included  in  the 
THPs  at  all. 

Video  Taped  After  Action  Review  tVAAR^ 


The  VAARs  were  prepared  by  the  TAF  Division  for  each  TF  mission.  The  VAARs  were  a 
supplement  to  the  THP  to  assist  future  home  station  training.  The  VAAR  considered  key  to 
training  effectiveness,  was  a  tactical  discussion  among  all  soldiers.  AARs  provided  a 
comprehensive  summary  of  the  battalion's  training  performance  and  were  the  most  immediate 
feedback  provided.  The  discussions  were  conducted  by  the  senior  trainer  (OC)  immediately 
following  each  mission  training  exercise  (Ford  &  Hamza,  1992;  Kemper,  1986).  VAARs  answered 
the  questions: 


'What  happened' 

'Why  it  happened' 

'How  to  fix  deficiencies' 

In  1986,  the  VAARs  were  examined  for  research  application  and  the  compliance  of  OCs  to 
the  prototype  VAAR  (Zimmerman,  Nichols,  &  Kemper,  1987).  While  substantive  information  was 
contained  on  the  tapes  (e.g.,  persistent  criticism  of  communications,  artillery,  and  fire  support 
accuracy),  the  OCs  did  not  follow  the  prototype  which  decreased  the  research  utility  of  the  tapes. 
Only  half  of  the  OCs  reviewed  the  training  objective  for  the  mission,  none  of  the  OCs  reviewed  the 
task  and  subtasks  from  the  ARTEPs,  and  only  a  few  used  the  checklists  created  to  standardize 
AARs  (Shackelford,  1985).  Platoon  OCs  almost  never  reviewed  the  TE&Os,  and  two-thirds  of  the 
Company  OCs  lectured  rather  than  guided  the  feedback  sessions.  At  the  TF  level,  OCs  followed  the 
prototype  more  closely,  but  still  rarely  summarized  events  or  results  in  relation  to  training 
objectives.  Currently,  it  is  recognized  that  OCs  conduct  feedback  under  the  rubric  of  BOS  at  all 
echelons.  Checklists  and  T&EOs  were  rarely  used  to  guide  feedback. 


At  each  level  of  the  brigade,  AARs  were  streamlined  to  focus  on  specific  tasks  and  training 
points.  This  allowed  analysts  to  use  the  AARs  in  conjunction  with  other  data  to  capture  a  more 
complete  picture  of  battlefield  events.  However,  as  an  independent  research  tool,  utility  was 
limited.  The  major  objectives  of  the  AAR  remained  generally  the  same  throughout  the  time  the 
CTC  Archive  remained  at  ARI-POM.  The  structure  of  the  AARs  were  organized  by  battle  phase 
(Plan,  Prepare,  Execute),  the  seven  BOSs,  and  key  events. 
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The  objectives  were  (Ford  &  Hamza,  1992): 

•Identify  and  highlight  action  and  events  by  each  primary  area  of  responsibility  critical  to 
the  mission  outcome. 

•Examine  course  of  actions  to  note  deviations  from  original  plans  using  a  participatory 
method  versus  lecture  for  generating  an  improved  course  of  action  to  follow  for  subsequent 
missions. 

•Analyze  the  effectiveness  of  major  BOSs  on  the  basis  of  critical  points. 

As  of  February  1994  (Ford  &  Hamza,  1992),  the  CTCs  conducted  routine  AARs  for 
Brigade,  Task  Force,  Field  Artillery,  Aviation,  and  the  Forward  Support  Battalion,  and  subordinate 
units.  AARs  continued  to  be  conducted  after  each  TF  mission  for  all  force-on-force  and  live  fire 
exercises,  of  which  there  were  8  or  9  missions  at  NTC,  5  missioiis  at  JRTC,  and  3  missions  at 
CMTC.  Company  and  platoon  AARs  were  conducted  in  the  field,  but  not  all  were  taped.  Special 
AARs  which  were  more  BOS  specific  were  conducted  by  the  Commander  of  the  Operations  Group 
(COG)  before  live  fire  exercises  for  each  TF.  Additional  AARs  were  conducted  for  TF  CSS,  Scout 
Platoons,  NBC,  Engineers,  and  a  Final  AAR  which  was  the  summation  of  all  missions. 

Operations  Plans 

The  Operations  Plan  is  a  planning  document  provided  by  the  NTC  to  the  TF  controlling 
brigade  to  be  executed  by  the  battalion  TFs  at  the  NTC.  Information  typical  of  the  Operations  Plan 
included: 


•Missions  that  will  be  executed 

•General  Astronomical  data  across  the  time  period 

•Scenario  Plan 

•Time/Event  Schedule  for  Mechanized  and  Armor  TFs,  and  Opposition  Force  (OPFOR) 

•General  Situation  narrative  describing  a  hostile  international  condition  with  potential  for 
war  between  the  two  super-powers 

•OPFOR  and  TF  with  General  Situation  Update(s) 

•Analysis  of  Area  Operations  narrative  with  a  General  Description  of  the  Area 
•Defense  Intelligence  Report  on  the  Handbook  Krasnovian  Army  (OPFOR) 

•Series  of  Operations  Orders  (OPORD),  Warning  Orders,  and  accompanying  overlay  maps 

Operations  Plans  were  received  on  paper  and  acetate.  When  technology  advanced  to  the 
point  where  computers  could  "read"  pictorial  data,  1989,  the  Operations  Plans  were  integrated  into 
the  digital  database  (Lewman,  Zimmerman,  Briscoe,  Root,  &  Alan,  1989).  Often  the  acetate  and 
paper  documents  were  too  large  to  be  "read"  directly  into  the  computer.  Hence,  an  individual  had 
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to  reduce  the  size  via  photocopy  machines,  which  involved  a  great  deal  of  cutting  and  pasting. 
Later,  circa  1991,  video  cameras  could  be  used  to  "scan"  the  entire  image  and  then  "read"  it  into  the 
computer. 

By  1995,  NTC  was  putting  the  Operations  Plans  and  other  graphics  into  the  computer  at  the 
TAP.  However,  no  software  existed  nor  was  implemented  which  would  permit  the  overlays  to  be 
viewed  in  conjimction  with  the  battle  replays.  At  the  time  the  CTC  Archive  was  turned  over  to 
CALL,  Ft.  Leavenworth,  efforts  had  been  under  way  to  join  the  computerized  graphics  with  battle 
replays  (Hutton,  1994). 

Table  12.  CTC  Archive  1995  Summary  of  Contents 


File  Type 

-  --  Files 

Size 

Automated  Finder's  Guide 

18 

584,898 

CMTC  Graphics  Files 

539 

44,954,862 

JRTC  Archive  Data 

1,128 

41,213,831 

JRTC  Graphics  Files 

308 

49,209,382 

JRTC  THP  Files 

830 

20,378,336 

Mission  Critical  Event  Sheet 

515 

875,938 

NTC  Archived  Data 

1,094 

48,967,078 

NTC  Graphics  Files 

1,777 

216,304,891 

NTC  Mission  Database  Files 

8,417 

1,862,946,174 

NTC  Replay  Files 

448 

63,423,780 

NTC  THP  Files 

4,493 

114,112,959 

Unit  After  Action  Reports  tUAARs^ 


The  UAARs  were  a  detailed  narrative  and  numerical  description  provided  by  the  brigade 
commander  to  the  division  commander  on  the  NTC  training  period.  Recommendations  for 
improving  the  NTC  experience  were  provided.  Basically,  the  UAARs  were  self-reports  which 
sometimes  contained  a  slightly  different  perspective  than  the  THPs  or  AARs  (Sulzen,  1995’). 
This  data  source  was  sent  to  ARJ-POM  in  the  early  period,  but  discontinued  shortly  thereafter.  The 
UAARs  included  (Kemper,  1986): 

•Narrative  Overview 

•Each  TF  Commander's  Comments 


’  Telephone  conversation  28  September  1995. 
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•Task  Organization 

•Specific  Comments  and  Recommendations  on  the  seven  BOS: 

-Command  &  Control 

-Maneuver 

-Fire  Support 

-Intelligence 

-Air  Defense 

-Mobility/Countermobility/Survivability  (Survivability  was  a  separate  BOS  during 
1987) 

-Combat  Service  Support 

•Numeric  Data  on: 

-Maintenance 
-Major  Assembly  Usage 
-Vehicle  Requirements 
-Budget 

Communications  Tapes  twith  CHOIS') 

Avant,  Kemper,  and  Henderson  (1986)  examined  the  tapes  to  determine  their  research 
utility.  The  obtaining  of  data  fi'om  this  source  was  labor  intensive.  The  recorded  information  does 
provide  useful  information  and  has  been  used  to  complement  research  efforts,  however  the 
extraction  of  the  communications  takes  a  great  deal  of  time.  The  tapes  consisted  of  40-channels,  all 
in  operation  simultaneously.  Related  documents  were  found  to  be  absolutely  necessary  to  make  use 
of  the  commo  tapes.  The  CEOIs  were  necessary  to  determine  what  channels  and  who  was 
designated  to  use  which  channels.  Commo  cut  sheets,  daily  commo  log  sheets,  and  tracking  lists 
for  both  BLUFOR  and  OPFOR  were  likewise  required.  The  machine  required  to  playback  the  tapes 
was  also  very  expensive.  Frequency  of  communications  was  available  through  the  digital  data,  so 
the  efficiency  of  this  effort  remained  dubious.  Researchers  sometimes  used  the  tapes  to  gain  a  more 
comprehensive  view,  but  the  effort  was  labor  intensive. 

In  1991  Mulshine  again  reviewed  the  tapes  for  research  applicability.  ARI-POM  facilitated 
its  use  by  recording  one  channel  at  a  time  and  eliminated  long  periods  of  silence.  This  was 
accomplished  by  a  computerized  program  which  turned  on  when  voices  were  engaged  and  off  when 
voices  were  silent  (analog  system).  The  cassettes  contain  a  digital  time  signal  on  one  channel,  with 
the  voice  on  the  other  channel;  thus  a  reseeircher  could  identify  when  a  transmission  was  occurring. 

Communications  tapes  are  a  potentially  rich  data  source  which  has  been  used  when  time 
permitted  and  the  research  indicated  its  usefulness  (Jarrett,  1994).  However,  communications  tapes 
will  only  be  useful  when  technology  is  such  that  voice  recognition,  in  combination  with 
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noise-filtering  can  transcribe  the  communications  to  a  time-tagged  ASCII  file  which  then  could  be 
combined  with  the  other  digital  databases  (Lewman,  et  al.,  1992). 
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CONCLUSION 

ARI-POM  developed  and  delivered  to  TRADOC  a  modem,  user-friendly,  largely  digital 
information  system  with  the  capability  to  derive  institutional  lessons  for  the  Army  from  training 
conducted  at  the  CTCs.  The  CTCs  generate  rich  sources  of  data  during  the  conduct  of  CTC  training. 
Battalions  have  been  performing  three  to  five  missions  each  rotation,  under  similar  conditions  for 
over  10  years.  Currently,  33  units  rotate  through  the  CTCs  yearly.  ARI-POM  successfully 
integrated  90%  of  the  divergent  data  sources  into  an  accessible  database  complete  with  tools  and 
programs  to  assist  analysts.  Military  analysts  and  researchers  are  now  able  to  derive  information 
remotely  or  on  site  using  the  tools  and  programs  ARI-POM  developed. 

Insights  useful  to  unit  trainers  are  important.  The  current  downsizing  in  force  and  budget 
means  training  dollars  are  also  constrained.  Additionally,  unit  post  rotation  turbulence  has  been 
often  as  high  as  20%  after  one  month,  including  primary  staff  positions.  After  three  months, 
turbulence  has  been  such  that  the  unit  conducting  home  station  training  was  no  longer  the  same 
unit  that  trained  together  at  a  CTC  (Chapman,  1992;  Keesling,  1993).  The  importance  of  deriving 
institutional  lessons  for  the  Army  is  more  relevant  now  than  ever.  The  CTC  Operational  Archive 
provides  the  tools  required  to  derive  needed  lessons. 

As  the  Army  modernizes  and  its  roles  perhaps  change,  the  CTC  Archive  has  the  potential 
to  play  a  crucial  role  in  deciding  which  'improvements'  contribute  to  the  Army's  ability  to 
perform  its  missions  and  indicate  when  they  do  not.  The  advent  of  Force  XXI  and  attendant  new 
systems  will  require  the  Army  to  learn  quickly  and  effectively.  The  CTC  Archive  provides  the 
ability  to  examine  the  utility  and  effectiveness  of  new  doctrine,  organizations,  training,  materiel, 
and  leadership. 

ARI-POM  introduced  methods  and  programs  with  which  to  conduct  studies  and 
performed  in-house  research  to  support  the  Army.  Doctrine,  organization,  training,  materiel,  and 
leadership  (DOTML)  were  assessed  with  data  processed  at  the  CTC  Archive.  The  CTC  Archive 
demonstrated  it  could  provide  needed  information  for  the  Army. 


"Those  who  cannot  remember  the  past  are  condemned  to  repeat  it. "  George  Santayana. 
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